- Joined
- Jan 8, 2019
- Messages
- 56,623
- Solutions
- 2
- Reputation
- 32
- Reaction score
- 100,456
- Points
- 2,313
- Credits
- 32,750
6 Years of Service
76%
rling is similar to the rli utility found in hashcat-utils, but much, much faster.
Technologies
- Dynamic sizing
- rling dynamically sizes the memory to be appropriate for the file. By not having compiled-in limits for things like line lengths for the input lines, users are able to focus on novel use cases for the program.
- Hashing with xxHash
- xxHash is a great new hashing method - its very fast and portable. By combining a dynamic (overridable) hash table with an excellent hash function, performance was accelerated greatly. Also, because hash tables are sized dynamically, there is no need to guess "optimal" hash sizes.
- Multi threaded Binary searches and sorts
- Thanks to blazer's multi threaded qsort, sorts are very fast, and make use of all of your system thread and multicore resources. In general, hashing is faster than binary search, but binary search uses half of the memory, and can be many times faster for certain kinds of input.
- Filesystem-based database for very large files
- If you need to process very large datasets, and don't have enough memory, the -f option allows you to use a Berkeley db-based database instead of in-memory. This does allow unlimited file sizes, but you do need substantial free disk space. Use the -M option to give it more cache for the database, and -T to tell it where to put the databases (defaults to current directory).
- Memory use estimates
- For large files, memory use can still be high. rling displays the estimated amount of memory to be used as soon as practical after reading the input files. This can still be "too late" for some use cases - in general, you need at least 2 times the input file size in memory.
- stdin/stdout/named pipes fully supported
- Thanks to the "read exactly once, write exactly once" method rling uses for file I/O, stdin/stdout and named pipes can be used in any position that requires a file name. This is great for creating complex workloads.