in 2022, I wrote cmdlets for tar and gzip in PowerShell

Table of Contents

1. "an illegible trial-and-error mess"

As I wrote at the time:

That is a chaotic mess of trial-end-error of trying to make not even just gzip but even the TarWrite alone work and it doesn't want to work too well so far. I'm giving the MemoryStream for the plain tar twice the all-files size sum initial capacity, and the MemoryStream for the gzip once ditto. I am filtering out the dotfiles, especially as for example .gitignore is not considered hidden by Get-ChildItem.

I am quite stuck at this time but I will see if I'll manage to break thru it further. 

$GemCap = "~/gemcap"
$TotalDirectorySize = (Get-ChildItem $GemCap -File -Recurse |
    Measure-Object -Sum Length |
    Select-Object -ExpandProperty Sum)
$TarStream = ([System.IO.MemoryStream]::new([int]$TotalDirectorySize * 2))
Add-Type -Assembly "system.formats.tar"
$TarWriter = ([System.Formats.Tar.TarWriter]::new($TarStream))

Add-Type -Assembly "system.io.compression.filesystem"
#$GzOut = ([System.IO.MemoryStream]::new($TotalDirectorySize))
$GzOut = [System.IO.File]::OpenWrite("C:\Users\m\helo.tar")
# $Gzipper = ([System.IO.Compression.GZipStream]::new(
    # $GzOut, [System.IO.Compression.CompressionLevel]3))
    #[System.IO.Stream]([System.IO.File]::OpenWrite( "C:\Users\m\helo.tgz" )), [int]81920
# $Promise = ($TarStream.CopyToAsync($Gzipper))
#$Promise = ($TarStream.CopyToAsync($GzOut))
Get-ChildItem $GemCap -File -Recurse -Name |
Where-Object { $_ -NotMatch "^(.*[\\\/])?\..*" } |
ForEach-Object { $TarWriter.WriteEntry( (Join-Path $GemCap $_ -Resolve),$_ ) }
#$Promise.Wait()
$TarStream.CopyTo($GzOut)

2. A month later I tried writing a cmdlet instead

2.1. A failed attempt at a cmdlet

Imports System.IO
Imports System.Management.Automation
Imports System.Formats.Tar
Imports System.IO.Compression

<Cmdlet("New", "TarGz")>
Public Class NewTarGzCommand
    Inherits Cmdlet
    <Parameter(ValueFromPipeline:=True, Mandatory:=True)>
    Public FileNames As String()
    <Parameter(Mandatory:=True, Position:=0)>
    Public RootPath As String

    Protected Overrides Async Sub ProcessRecord()
        Dim filePaths = From fileName In FileNames
                        Select New With {
                .PathFileName = Path.Join(RootPath, fileName),
                .EntryName = fileName}

        Dim totalDirectorySize = (From filePath In filePaths
                                  Select FileLen(filePath.PathFileName)
                ).Sum()
        Dim tarStream = New MemoryStream(totalDirectorySize * 2)
        Dim tarWriter = New TarWriter(tarStream)
        Dim gzOut = New MemoryStream(totalDirectorySize)
        Dim gZipper = New GZipStream(gzOut, CompressionLevel.SmallestSize)
        Dim tarToGzPromise = tarStream.CopyToAsync(gZipper)
        For Each f In filePaths
            tarWriter.WriteEntry(fileName:=f.PathFileName, entryName:=f.EntryName)
        Next
        tarWriter.Dispose()
        Await tarToGzPromise
        WriteObject(gzOut.ToArray(), True)
    End Sub
End Class
And the Powershell script to use it:
$GemCap = "~/gemcap"
Get-ChildItem $GemCap -File -Recurse -Name |
Where-Object { $_ -NotMatch "^(.*[\\\/])?\..*" } |
New-TarGz (Resolve-Path $GemCap)

But it didn't work:

And it doesn't work!

The script returns no output, redirecting to file results in an empty file

Time to split things up

I shouldn't really make a cmdlet that does both tar and gz because it won't be modular enough. I dislike how Powershell piping here apparently can only do sending whole collections after they are finished, but here we aren't going to have the archives too large (we're storing them in byte arrays anyway, after all), so maybe it would be good to write two separate cmdlets.

2.2. An attempt at a tar cmdlet

Imports System.IO
Imports System.Management.Automation
Imports System.Formats.Tar

<Cmdlet("New", "Tar")>
Public Class NewTarCommand
    Inherits Cmdlet
    <Parameter(ValueFromPipeline:=True, Mandatory:=True)>
    Public FileNames As String()
    <Parameter(Mandatory:=True, Position:=0)>
    Public RootPath As String

    Protected Overrides Sub ProcessRecord()
        Dim filePaths = From fileName In FileNames
                        Select New With {
                .PathFileName = Path.Join(RootPath, fileName),
                .EntryName = fileName}

        Dim totalDirectorySize = (From filePath In filePaths
                                  Select FileLen(filePath.PathFileName)
                ).Sum()
        Dim tarStream = New MemoryStream(totalDirectorySize * 2)
        Dim tarWriter = New TarWriter(tarStream)
        For Each f In filePaths
            tarWriter.WriteEntry(fileName:=f.PathFileName, entryName:=f.EntryName)
        Next
        tarWriter.Dispose()
        WriteObject(tarStream.ToArray(), True)
    End Sub
End Class

2.3. How do I redirect binary in Powershell?!

Running the above with 

Get-ChildItem $GemCap -File -Recurse -Name |
 Where-Object { $_ -NotMatch "^(.*[\\\/])?\..*" } |
 New-Tar (Resolve-Path $GemCap) > examplefilename.tar

We get a file.

A text file. A list of decimal byte values. Our return type was an array of bytes.

2.3.1. Solution

People say the ways are 

[io.file]::WriteAllBytes(filename, $_)
Set-Content -Path path -AsByteStream

This is laughable. Modern Powershell really has no better ways?! But ok, I used the latter one for now.

2.4. First result - tar format issues

The result turns out to be tar file with hidden contents - bsdtar 3.5.2 as well as 7-Zip only see the first file added, but I can see all of the files' contents when inspecting it raw with notepad. A 660KB tar file in which only the first added 4KB file is found.

"PaxHeaders" can be found repeatedly in files' headers, so that indicates WriteTar/WriteEntry defaulted to the "Pax" POSIX tar format mentioned as one of the options in its documentation. So I decided to initialize TarWriter to another format: 

New TarWriter(tarStream, TarEntryFormat.Ustar)
> Ustar 	2 	POSIX IEEE 1003.1-1988 Unix Standard tar entry format.

The result: notepad inspection shows that the format is much different, but the result in bsdtar -t and in 7-Zip is the same. I even checked bsdtar -xvp just to be sure. And the filenames of all the "hidden" files are also still there in the file!

I would also like to mention that just in case, I don't really have a way at hand to debug these DLLs as i Import-Module them in Powershell.

Should I do another attempt with `TarEntryFormat.Gnu`? Ughhh should I.. I lazily did, it seemed undistinguishable from the old file that i never checksummed, there is a chance i just looked at the old file or a result of the old DLL.

No idea where to go from there. But ok, I got that bsdtar at C:\Windows\system32\tar.exe I should probably go on to use it and make a gzipping cmdlet

2.4.1. How the processing in a Powershell cmdlet really works

> The Process statement list runs one time for each object in the pipeline. While the Process block is running, each pipeline object is assigned to the $_ automatic variable, one pipeline object at a time.

The above is an excerpt from about_Functions, as I wanted to start writing my stuff in Powershell again, so I read up both about creating Cmdlets with about_Functions_Advanced as well as about regular functions, and decided to use the regular functions this time. The whole reason to stray from pure Powershell into .NET was not being able to do some too dotNETy things revolving around streams, but now it appears I no longer need to safe-haven in it for things to feel sane enough.

I was actually very dum seeing that I'm overriding a `ProcessRecord` method and not thinking any of it. I had the feeling that something may ultimately be off when I don't have debugging for the DLLs that i Import-Module, but decided it couldn't. Now I finally know roughly what those mysterious methods to override for Begin and End do.

How the heck did all the filenames go into the tar, though? My understandings of things clash so badly now. I gotta make another take at the Tar cmdlet.

Ooh, maybe because it were actually many tars appended to each other. I guess it's not too easy to distinguish multiple tars appended to each other from a single-made tars.

2.4.2. This New-Tar produces valid tars! I did not improve further.

Imports System.IO
Imports System.Management.Automation
Imports System.Formats.Tar

<Cmdlet("New", "Tar")>
Public Class NewTarCommand
    Inherits Cmdlet
    <Parameter(ValueFromPipeline:=True, Mandatory:=True)>
    Public FileName As String
    <Parameter(Mandatory:=True, Position:=0)>
    Public RootPath As String
    Public Shared MEMORYSTREAM_KB = 1000
    Private Shared ReadOnly Property MemoryStreamCapacity As Int32
        Get
            Return MEMORYSTREAM_KB * 1024
        End Get
    End Property
    Private tarStream = New MemoryStream(MemoryStreamCapacity)
    Private tarWriter = New TarWriter(tarStream, TarEntryFormat.Ustar)

    Protected Overrides Sub ProcessRecord()
        tarWriter.WriteEntry(
            fileName:=Path.Join(RootPath, FileName),
            entryName:=FileName)
    End Sub
    Protected Overrides Sub EndProcessing()
        tarWriter.Dispose
        Dim tarArrayResult As Byte() = tarStream.ToArray
        WriteObject(tarArrayResult, True)
    End Sub
End Class

It now produces valid tars!

With just one issue, the backslashes in paths became middle-square-dot (``) characters.

Managed to get that fixed with adding a dumb character replace to the pipe: 

| foreach-Object { $_ -replace '\\', '/' } |

These tars seem to be fine.

2.5. Compress-Gzip cmdlet

Imports System.IO
Imports System.IO.Compression
Imports System.Management.Automation
<Cmdlet("Compress", "Gzip")>
Public Class CompressGzipCommand
    Inherits Cmdlet

    <Parameter(ValueFromPipeline:=True, Mandatory:=True)>
    Public Input As Byte

    Public Shared MEMORYSTREAM_KB = 1000
    Private Shared ReadOnly Property MemoryStreamCapacity As Int32
        Get
            Return MEMORYSTREAM_KB * 1024
        End Get
    End Property

    Private ReadOnly result = New MemoryStream(MemoryStreamCapacity)
    Private ReadOnly zipper = New GZipStream(result, CompressionLevel.SmallestSize)

    Protected Overrides Sub ProcessRecord()
        zipper.WriteByte(Input)
    End Sub

    Protected Overrides Sub EndProcessing()
        zipper.Dispose()
        Dim resultBytes As Byte() = result.ToArray
        WriteObject(resultBytes)
    End Sub

End Class

invoked 

Get-Content -AsByteStream -Path .\file.tar | Compress-Gzip | Set-Content -AsByteStream -path .\file.tar.gz

does produce a valid tar.gz accepted by tar xz.

Yeah it does work it byte by byte because I wasn't sure what the behavior could be with multiple byte arrays being processed. But doesn't seem too bad and at least is streaming.

2.6. Usage

Get-ChildItem $GemCap -File -Recurse -Name |
 Where-Object { $_ -NotMatch "^(.*[\\\/])?\..*" } |
 foreach-Object { $_ -replace '\\', '/' } |
 New-Tar (Resolve-Path $GemCap) | Compress-Gzip | set-content -path realshit.tar.gz -asbytestrea