Difference: FortranBase64 (1 vs. 4)

Revision 413 Jun 2006 - FawziMohamed

Line: 1 to 1
 
META TOPICPARENT name="ProjectPage"

Fortran 90 base64 encoding of Reals

Line: 26 to 26
 In the directory there is a fortran file for the tool (convertBase64.F90) and also some supporting files to test it, in particular the python script =testBase64.F90. If you have problems on some architectures/compilers try running it to pinpoint the problem.

I think that the code is close to optimal from the feature point of view.

Deleted:
<
<

Problems

What are the problems
 
Changed:
<
<
-- FawziMohamed - 24 May 2006
>
>

Problems

Unfortunately in all the compilers that I came across formatted write/read is much less optimized than binary, and sting handling is not efficient. Base64 is faster than a formatted write, but slower than binary. If formatted write was an option, base64 is a clear winner, but if you need top performance writing to a local disk then it might not be for you.

Benchmark

Done with an older version, the actual version is 10-20% faster for base64. Reading is similar. Opteron writing to a local disk:

20x1e6 reals:
intel, binary: 0.17s total, 20x7.8MB
intel, base64: 6.54s total, 38% strcopy, 48% write, 20x10.4MB
intel formatted write: 37.7s total, 78% in sys calls (write whole array at once) 20x34MB

20x1e7 reals:
intel, binary 1.5s total, all in system calls, 20x78MB
intel, base64 71s total, 34% strcopy, 45% formatted write (530 characters at once) 20x104MB
intel, formatted write 364s total, 78% in systems calls (write whole array at once) 20x346MB

20x1e8 reals:
intel, binary 16.4s total , (31s elapsed), all in system calls 20x782MB
intel, base64 650s total, 47% formatted write, 32% strcopy , 20x 1047MB
intel formatted not done

-- FawziMohamed - 13 Jun 2006

 

Revision 313 Jun 2006 - FawziMohamed

Line: 1 to 1
 
META TOPICPARENT name="ProjectPage"

Fortran 90 base64 encoding of Reals

Line: 26 to 26
 In the directory there is a fortran file for the tool (convertBase64.F90) and also some supporting files to test it, in particular the python script =testBase64.F90. If you have problems on some architectures/compilers try running it to pinpoint the problem.

I think that the code is close to optimal from the feature point of view.

Added:
>
>

Problems

What are the problems
  -- FawziMohamed - 24 May 2006

Revision 224 May 2006 - FawziMohamed

Line: 1 to 1
 
META TOPICPARENT name="ProjectPage"

Fortran 90 base64 encoding of Reals

Line: 7 to 7
 This encodes 6 bits (64 values) in one char (8 bits) using "A-Za-z0-9+/", which means that it is 1/3 larger than binary encoding. If done well it is fully portable, and doable in fortran without external libraries.
Deleted:
<
<
So I tried to implement it, and it was much harder than I thought to do it in a portable way across compilers.
 This code has the ability to encode and decode reals to/from base64 even across precision (i.e. decode single precision to double and double precision to single), and to handle infinity and NAN consistently.

In base64 an array of reals is represented like this:

Changed:
<
<
>
>
<reals bit_mantissa=23 bit_exp=8 dims="10 6 8">
 Y4jImSka/AEAAAAAAAAA4AAAAAAAAA9NAAAAAAAAQgD ...
Changed:
<
<
>
>
</reals>
  Bit mantissa and exponent are stored so that the precision of the model is known and transfer between precisions is possible.
Changed:
<
<
CompilerWoes
>
>
I wrote a small utility to convert to/from base64, so that you can get the data in/out easily. You can find it in cp2k/tools/base64.

It was much harder than I thought to make it work across the compilers, see CompilerWoes for a small description of the problems.

 
Changed:
<
<
I wrote a small utility to convert to/from base64, so that you can get the data in/out cp2k easily. You can find it in cp2k/tools/convertBase64.F90 (define __NAG, __XLF with the nag or xlf compilers). There is also a python script cp2k/tools/testBase64.py that if you feed the executable tests it. I would like to know the results on other architectures/compilers.
>
>
In the directory there is a fortran file for the tool (convertBase64.F90) and also some supporting files to test it, in particular the python script =testBase64.F90. If you have problems on some architectures/compilers try running it to pinpoint the problem.
  I think that the code is close to optimal from the feature point of view.

Revision 124 May 2006 - FawziMohamed

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="ProjectPage"

Fortran 90 base64 encoding of Reals

To store big amounts of reals encoding the IEEE754 bit pattern in big endian order to base64 is a good solution.

This encodes 6 bits (64 values) in one char (8 bits) using "A-Za-z0-9+/", which means that it is 1/3 larger than binary encoding. If done well it is fully portable, and doable in fortran without external libraries.

So I tried to implement it, and it was much harder than I thought to do it in a portable way across compilers.

This code has the ability to encode and decode reals to/from base64 even across precision (i.e. decode single precision to double and double precision to single), and to handle infinity and NAN consistently.

In base64 an array of reals is represented like this: Y4jImSka/AEAAAAAAAAA4AAAAAAAAA9NAAAAAAAAQgD ...

Bit mantissa and exponent are stored so that the precision of the model is known and transfer between precisions is possible.

CompilerWoes

I wrote a small utility to convert to/from base64, so that you can get the data in/out cp2k easily. You can find it in cp2k/tools/convertBase64.F90 (define __NAG, __XLF with the nag or xlf compilers). There is also a python script cp2k/tools/testBase64.py that if you feed the executable tests it. I would like to know the results on other architectures/compilers.

I think that the code is close to optimal from the feature point of view.

-- FawziMohamed - 24 May 2006

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback