This is a comparison of the support various languages and tools
have for reading and processing arbitrarily long lines of text.
This gives an idea of the base overhead of using these tools as filters.
I originally ran these tests over 6 years ago, and it's interesting to compare those results now. All results have improved of course in line with the better hardware, but awk, python and tcl have made additional notable improvements over the versions available in 2002.
Each program loops reading a line from stdin and outputting immediately to stdout, and was tested/timed using:time ./test_prog <test_data.txt >/dev/nullThere were 2 test files each containing 354371 lines of text. The short lines (/usr/share/dict/words) average line length was 8 characters (excluding the LF) and long lines (/usr/share/doc/*) where the average line length was 37 chars. Many runs were done and the average taken so that no disk access was involved etc.
--------------------------------------------------------------------------------------------- language/tool short lines (~8 chars) long lines (~37 chars) --------------------------------------------------------------------------------------------- C (getline) 0.082s (0.072s/0.003s) 0.110s (0.092s/0.012s) C (linebuffer getc unlocked) 0.067s (0.060s/0.001s) 0.138s (0.117s/0.015s) C (linebuffer fgets NoNulls unlocked) 0.109s (0.097s/0.006s) 0.162s (0.145s/0.011s) C (linebuffer fgets unlocked) 0.160s (0.144s/0.003s) 0.221s (0.193s/0.011s) C (linebuffer fgets NoNulls) 0.137s (0.120s/0.001s) 0.189s (0.163s/0.013s) C (linebuffer fgets) 0.178s (0.161s/0.003s) 0.233s (0.209s/0.012s) C (linebuffer getc) 0.067s (0.057s/0.004s) 0.137s (0.122s/0.009s) grep (2.5.1) 0.573s (0.557s/0.003s) 1.496s (1.469s/0.014s) cgrep (2.5.1) 0.109s (0.098s/0.006s) 0.135s (0.120s/0.010s) sed (4.1.5) 0.176s (0.163s/0.006s) 0.195s (0.176s/0.013s) perl (5.8.8) 0.371s (0.220s/0.008s) 0.337s (0.279s/0.011s) awk (3.1.5) 0.164s (0.149s/0.003s) 0.195s (0.174s/0.008s) cpp 0.741s (0.616s/0.119s) 2.145s (1.842s/0.134s) python (2.5.1) 0.746s (0.734s/0.006s) 0.825s (0.803s/0.014s) bash (3.2.25) 12.769s (11.230s/1.193s) 27.956s (24.836s/2.280s) dash (0.5.4) 3.899s (1.616s/2.275s) 14.891s (4.569s/9.813s) tcl (8.4.15) 2.361s (1.821s/0.176s) 2.375s (2.005s/0.194s) cat* 0.004s (0.000s/0.004s) 0.014s (0.001s/0.013s) --------------------------------------------------------------------------------------------- *cat actually does essentially the same since we're not actually changing the data :-) however it does illustrate the advantages of using the optimum device block size and bypassing stdio buffering etc. You can download all source code used for the test. Test host info: --------------- System 00:00.0 Host bridge: Intel Corporation Mobile 915GM/PM/GMS/910GML Express Processor to DRAM Controller (rev 03) CPU vendor_id : GenuineIntel cpu family : 6 model : 13 model name : Intel(R) Pentium(R) M processor 1.70GHz stepping : 8 cpu MHz : 1700.000 cache size : 2048 KB Kernel Linux 2.6.24.5-85.fc8 #1 SMP Sat Apr 19 12:39:34 EDT 2008 GLIBC VERSION="2.7" CFLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 \ -m32 -march=i686 -mtune=generic -fasynchronous-unwind-tables" GCC VERSION="4.1.2" CFLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 \ -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables"
© May 9 2008