Recently we had our unicode upgrade along with Ehp7. One of our batch job, which is used for uploading mass data from a file in application server to SAP system, intermittently starting giving runtime error.
Exception : CX_SY_CONVERSION_CODEPAGE
Reason: During conversion of a text from code page ‘4110‘ to code page ‘4102‘, one of the following occurred:
– characters were discovered that cannot be dislpayed in one of the code pages, or
– the conversion could not be performed for some other reason
The dump looked like below.
The dump took place at READ statement.
The issue was strange, as it worked for 99% of files but dumped for some very few. It means, the code works good for maximum cases but could not handle some specific scenarios. I looked at the text file which was loaded to application server very minutely. But, nothing suspicious was visible to open eyes.
Since most of the files were loaded successfully, I planned to investigate this file more to figure out what was different in this file from rest which loaded correctly. To my good luck, this sample file had just 3 lines, for me to scan through. But still, I did not find anything wrong.
Not sure why, may be flash of Mr Holmes , I thought of opening this file using Microsoft Excel. Opened a blank excel file. File -> Open -> txt file.
The moment I hit ‘Open’, look what I found!! Special characters treasure which I was hunting for. Somehow, the creator of this file ended up putting some special characters (may be Chinese) which our program was not able to handle.
Once you know the root cause, finding solution was no brainer. Our business analyst quickly deleted the special characters and re-ran the job. Boom!! everything loaded successfully.
As an ABAPer, I thought, our program should have been smart enough to handle such simple exception. I, did an ‘F4‘ on ‘OPEN DATASET‘ key words and found the below documents. I must have seen it hundreds of time, but for the first time I went through each words to understand what it meant. I wanted to understand, how the system would behave when we incorporate the key words specified in sap document.
I tried explicitly specifying UTF-8, but got the same dump.
OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING UTF-8 SKIPPING BYTE-ORDER MARK.
Then I tried replacing ‘ENCODING DEFAULT’ by ‘ENCODING NON-UNICODE‘.
OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING NON-UNICODE.
Bingo.. It worked!!!
Debug screen shows that SAP considers the special characters as space. So, no issue.
Is this a right way? I was not sure and I am not sure till date. May be some expert in this area can throw some light. What I could do best was run Extended Syntax Check and Code Inspector. None of them gave any warning or error. Hopefully SAP likes it.. 🙂
But already being in UNICODE system, is it good practice to OPEN file using NON-UNICODE encoding? Not sure.. 🙁
I also tried adding the key word ‘IGNORING CONVERSION ERRORS‘.
OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING DEFAULT IGNORING CONVERSION ERRORS.
The file did not dump in this case as well. SAP handled the conversion.
Debug screen shows that the special character is read as ‘#’ at runtime. Also note that the special character ‘#’ and the Tab separator ‘#’ looks the same.
Now, I was wondering, will the SPLIT AT TAB (separator) statement give wrong output. Look how it behaves in Debug.
For now, I believe Alternative 2 is OK, though I am not fully aware if we should use it or not.
If you can correct your file i.e remove unwanted characters and load them, that is even better. The best alternative
By any chance, have encounter this issue earlier? Please share how you resolved it. If you want to add some more information on this topic, please feel free to email me at firstname.lastname@example.org or leave your explanation in the comment section.
Till, someone provides the right answer with good justification, the mystery continues… 🙂
Sample text file with the special characters.
Code snippet used for analyzing the above scenarios.
If you have 5 minutes, you can upload the above text file to application server using t-code CG3Z and then create the program using the code snippet above and test it.
If you liked this page, you might like to check our other post on Unwanted Error.
Please subscribe to get updates about our new post.
Thank you very much for your time!!
Image source : www.flickr.com