Playing Sherlock Holmes to detect CONVT_CODEPAGE runtime error mystery

Sapyard.com
Share on Facebook7Share on LinkedIn10Tweet about this on TwitterShare on Google+0
Please Share!!

[adToAppearHere]

Recently we had our unicode upgrade along with Ehp7. One of our batch job, which is used for uploading mass data from a file in application server to SAP system, intermittently starting giving runtime error.

Exception : CX_SY_CONVERSION_CODEPAGE

Reason: During conversion of a text from code page ‘4110‘ to code page ‘4102‘, one of the following occurred:
– characters were discovered that cannot be dislpayed in one of the code pages, or
– the conversion could not be performed for some other reason

The dump looked like below.

4110 to codepage 4102

The dump took place at READ statement.

CONVT_CODEPAGE Conversion error
The issue was strange, as it worked for 99% of files but dumped for some very few. It means, the code works good for maximum cases but could not handle some specific scenarios. I looked at the text file which was loaded to application server very minutely. But, nothing suspicious was visible to open eyes.

CX_SY_CONVERSION_CODEPAGE

Since most of the files were loaded successfully, I planned to investigate this file more to figure out what was different in this file from rest which loaded correctly. To my good luck, this sample file had just 3 lines, for me to scan through. But still, I did not find anything wrong.

Not sure why, may be flash of Mr Holmes , I thought of opening this file using Microsoft Excel. Opened a blank excel file. File -> Open -> txt file.

4

The moment I hit ‘Open’, look what I found!! Special characters treasure which I was hunting for. Somehow, the creator of this file ended up putting some special characters (may be Chinese) which our program was not able to handle.

Unknown characters

Once you know the root cause, finding solution was no brainer. Our business analyst quickly deleted the special characters and re-ran the job. Boom!! everything loaded successfully.

As an ABAPer, I thought, our program should have been smart enough to handle such simple exception. I, did an ‘F4‘ on ‘OPEN DATASET‘ key words and found the below documents. I must have seen it hundreds of time, but for the first time I went through each words to understand what it meant. I wanted to understand, how the system would behave when we incorporate the key words specified in sap document.

6

Alternative 1.
I tried explicitly specifying UTF-8, but got the same dump.
OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING UTF-8 SKIPPING BYTE-ORDER MARK.

Alternative 2.
Then I tried replacing ‘ENCODING DEFAULT’ by ‘ENCODING NON-UNICODE‘.
OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING NON-UNICODE.

Bingo.. It worked!!!

8

Debug screen shows that SAP considers the special characters as space. So, no issue.
Is this a right way? I was not sure and I am not sure till date. May be some expert in this area can throw some light. What I could do best was run Extended Syntax Check and Code Inspector. None of them gave any warning or error. Hopefully SAP likes it.. 🙂

7

But already being in UNICODE system, is it good practice to OPEN file using NON-UNICODE encoding? Not sure.. 🙁

Alternative 3.
I also tried adding the key word ‘IGNORING CONVERSION ERRORS‘.
OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING DEFAULT IGNORING CONVERSION ERRORS.

The file did not dump in this case as well. SAP handled the conversion.

Debugging

Debug screen shows that the special character is read as ‘#’ at runtime. Also note that the special character ‘#’ and the Tab separator ‘#’ looks the same.

Now, I was wondering, will the SPLIT AT TAB (separator) statement give wrong output. Look how it behaves in Debug.

10
Although it did not dump, but the IGNORING CONVERSION ERRORS would add those unwanted ‘#’, if we load the data to SAP System.

For now, I believe Alternative 2 is OK, though I am not fully aware if we should use it or not.

If you can correct your file i.e remove unwanted characters and load them, that is even better. The best alternative

By any chance, have encounter this issue earlier? Please share how you resolved it. If you want to add some more information on this topic, please feel free to email me at mailsapyard@gmail.com or leave your explanation in the comment section.

Till, someone provides the right answer with good justification, the mystery continues… 🙂

Sample text file with the special characters.

Code snippet used for analyzing the above scenarios.

If you have 5 minutes, you can upload the above text file to application server using t-code CG3Z and then create the program using the code snippet above and test it.

If you liked this page, you might like to check our other post on Unwanted Error.

Please subscribe to get updates about our new post.

Thank you very much for your time!!

 

 

Image source : www.flickr.com

Share on Facebook7Share on LinkedIn10Tweet about this on TwitterShare on Google+0
Please Share!!

About the Author

SAP Yard
SAP Yard
SAPYard is one stop page for all Technical Folks in SAP. You would find un-conventional explanations, tutorials, and tricks. Please like our Facebook Page and also join our LinkedIn Group.

8 Comments on "Playing Sherlock Holmes to detect CONVT_CODEPAGE runtime error mystery"

  1. Thank You Raju,

    These has given me a way to resolve the issue. No one has identified this way to address this issue.

    Thanks Again.

    • Dear Anil – Thank you very much for your feedback. Glad that you appreciate our way of presentation. 🙂

      Please keep visiting and leaving your feedback.

      Regards,
      Team SAPYard.

  2. I don’t think this is the right approach. How do you know that what SAP skips is not of any importance ?

    In this case it wasn’t (Chinese characters ?), but it might as well be important to you. You will never know with your solution.

    I would ALWAYS want to download ALL characters without a dump. Then in another stage you can decide if the contents of the file were wrong or not.
    Option 2 (NON UNICODE) is therefore unacceptable for me.

    I would go down investigate the root cause, which is the encoding of the file on the server. I would check the file in notepad++ or any other texteditor to find out the character encoding.

    If you are on a unicode system then you should use encoding UTF-8.
    The problem therefore is not with SAP but with the application that created the file in the first place. It has been created in ANSI format. Even that I doubt. It looks it has been created in unicode first and then later converted to ANSI, thereby creating these unwanted characters.

    An alternative is to use notepad++ option convert to UTF-8 before loading it. But best is, in my opinion, to make sure that the application or person who creates this file is doing it in the right encoding.
    We had the same problem when certain names were containing special characters e.g. ğ or ç.
    If I would follow your example then these characters will not come into SAP, but I need them. So then my problem will still exist. I might have solved the dump, but I created another problem.

    • Dear Peter,

      Thank you so much for the detailed explanation. You have raised the valid point. Even I had that fear of omitting some valid characters. Therefore, I was expecting some expert to shower some more light.

      Agree, the best way is to get the files created in the right encoding.

      I will try to check in notepad++ .

      I will update the post with your inputs.

      Thanks again.
      Raju.

  3. It’s really useful!
    As you said, ” I must have seen it hundreds of time, but for the first time I went through each words to understand what it meant” – so do i. 😀

    • Dear Le Van – Glad that you think it is useful. We generally glance through the SAP document, with out analysing what it actually means.. Whenever we are in trouble, we tend to read more carefully.. 🙂 .. Glad that I am not the only one.. 😛

      Regards,
      Raju.

  4. Mutero william | August 17, 2015 at 8:42 am | Reply

    Thank you so much for sharing this with me. I have never come across that error but am sure have learn and if i get some infor will definitely share with you as well. Thanks for such helpful information. I keep learning from you every day.

    Regards

    William Mutero

Leave a comment

Your email address will not be published.


*