Project Plan Stage 2

This is the stage 2 documentation for project.

Build:

I was able to build the Zstandard software on the yaggi server with out any issues. I installed it locally to my directory just to play it safe as far as messing up the system or other files goes.
I tested several times with the software to compress my own files and it worked just fine.

The software did pass it's smoke test! It came with a make check file to run and it yield clean positive results.

As well the target system is the yaggi server which is an  x86_64 architecture server.

Test Setup:

As far as my test data goes I have collected a few screen shots that I had sitting on my phone as the test data. A selection of 15 photos. I plan to multiply that quantity by 10 giving me 150 photos to process which should and has shown a reasonable time to compress.  I'm simply just copying the same 15 files 10 times over for the larger quantity of test data. This way I can know it is at least consistent.

Benchmarks:

For the benchmarks I was able to  clock about 0.45 seconds to compress 150 photos (which is super quick - good job Facebook). I ran theses tests well over 10 times both with the small 15 sample size and the 150 sample size. It simple just multiplied the time it took by 10. Which is what I was expecting. That showed to me that there was nothing fancy happening with the larger amount of data. Just a static system for that data set multiplied. So this was very repeatable and you should yield the same results within reasonable times just due to the nature of server loads. Now out side of the times, I ran perf and the and found out that with the 15 photo sample the compression took less time than the overhead of the system setting things up. When I switched to the 150 sample the opposite happened. Which makes sense simply due to it taking 10 times longer to compress with the data being 10 times larger.  The function was actually a constructor. As see the "compressBlock_doublefast"  That is the main compression system doing the heavy lifting for the program. The rest of the red in the photo below you see  is mostly just system set up! This was super consistent across multiple tests,  even after warming up the cache.

Project Update

Sofar I was having the issue of the server not having the hardware needed to run perf. That was recently resolved last night. So I finally went ahead and ran some of the perf tests on the Zstandard software running it against a pack of varied sized jpeg photos. a set of 15, and then set of 150. ( I made 10 copied of each photo) I wanted to see consistency.  I also ran a few time tests on it and I got it with in a few thousandth of a second as far as how close to similar results go. The time just scaled by a factor when I multiplied the quantity of photos by 10. It simply took 10x longer. Which made sense. It was the same data but 10 times more.  Not the perf. I found out that the time to compress 15 photos was so small that the system overhead setting up the program took 5 times longer than the compression! How ever when I bumped the number of photos up to 150, that obviously dropped. So it was roughly  50% to compression and about 15-30% for system overhead.


This is the time test with 15 photos


This is the time test with 150 photos


This is the perf report compressing 150 photos! You can see it spends a lot of time on the one function. Which is actually just a constructor for an object.  The file running it has a lot of checking going on though. different file sizes, finding files, checking end of streams and such.


And lastly this is the assembler behind that main hot function. This is what the perf yields when you go deeper into the graph to look at what is happening. You an see that a  cmp in the assembler is being called a lot.

As of now I still do not have 100% of what I want to do with the project plan, but It should all be written up tomorrow night.

Lab 5

In this  lab we ran through a few algorithms for volume adjustment.
We used a sample size of 100 million for the tests.

In our first test we ran the multiplied the samples by  0.5

In the second test we precalculated the samples before modifying the volume.

Lastly in the third test we multiplied the samples with an int and then bit shifted the samples to the appropriate values.

For the results:

Test 1: This seemed to have run at a middle speed being processor heavy with calculations
Test 2: This test ran the slowest as it took more cache and memory to process the large amount of samples
Test 3: Lastly this test ran the fastest. It was processor heavy but bit shifting seemed to save it some time.

Project Plan

The Package?
I'll be looking into the ZStandard compressor. It is supposed to be a higher speed compression software.  I have yet to be able to get a proper perf command to run on it so I'll be looking more into that as soon as I can. But the compression seems like It should be able to have some sort of modification done to either improve size or speed so that is what I am aiming for:

Testing/Data?

 I plan to use a variety of  photos and text files to and compress them all at the same time and separately. Use them as my bench marks to see how it can handle the different types of data.  This should allow for a better in depth look at which parts are used in the compression


Optimizing?

I'd first like to try compiler options to see if there is any obvious improvements or breaking of the program.  Things like setting to O3, enable vectorizing.. Additionally if the compiler settings do nothing or worsen it I will dig into the compression or decompression where hot spots show up to see what can be done. Again I have yet to be able to check due to server issues but that's the goal.

Struggles of finding a package!

So I spend a good few hours the other night trying to find a package in a panic to not only find one not taken by other students, one that was in c/cpp,  and then trying to find something inside of the package that might yield some results of promise for the project. I personally couldn't really find what I thought that I might need to be looking for and was just confused and lost. So I ended the night and planned to try again the next night. I looked at PeaZip, and Zstandard. PeaZip was after a while of trying to get the files I found it that It was pearl. Which I probably could have guessed from the "Pea" part of the name.  Zstandard was in c /cpp which was great but I was starting to get tired and stressed to and at that point had minimal to no idea what I was trying to accomplish. So i'll be running test on Zstandard tonight to test it and find out what I might be able to do to it.