How can I add \n(newline) In Json string

Harsh · September 19, 2022, 10:05am

I want to do something like this.
data = '{JsonString}\n'.
The \n should be considered as a json newline.
I want to then send this json along with requests.post() method.
Thanks.

vbrozik · September 19, 2022, 2:09pm

I do not understand what you mean by “json newline”. JSON is a text format which allows you to store data. You can use the newline (ASCII 0xa) character in JSON as a whitespace character.

In Python '\n' is the correct way to represent a newline in a string. So for example this assigns a correct JSON string to the variable data:

data = '{"firstName": "John"}\n'

Harsh · September 20, 2022, 5:50am

Thanks for your reply.

**{'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'The msearch request must be terminated by a newline [\n]'}], 'type': 'illegal_argument_exception', 'reason': 'The msearch request must be terminated by a newline [\n]'}, 'status': 400} python**

When I am trying as you suggested I am getting above mentioned error in response of post method.

data = '{"firstName": "John"}\n'

This did not work for me.

vbrozik · September 20, 2022, 11:44am

OK, this is an error message of the server. It has nothing to do with Python. You need to learn about what exactly the API expects in your request.

The text I have shown to you was just an example. I have no idea what your server accepts as requests.

In my example the trailing newline is just a whitespace which does not change the data inside JSON. If you need to have a newline inside a string inside JSON you can use the two literal characters \n which means you will need to double the backslash to tell Python to produce a single regular backslash (instead of treating it as a special character).

Let’s take my example and add \n to the property value of firstName:

data = '{"firstName": "John\\n"}\n'
#                               ^^--- These two characters create a newline
#                                     character in the resulting string.
#                          ^^--- The two backslashes create a single literal
#                                literal backslash \ in the resulting string.
#                                The characters \n are then interpreted in JSON
#                                as a newline character.

Harsh · September 21, 2022, 7:17am

API expects the json to be terminated by newline. I did the same API call using Postman in that I had only put newline at the end of json by pressing ‘enter’ key. In that case It worked fine & proper response was received but while using python requests im passing the same json as a string. I tried to give \n in the end as you suggested but I’m getting error there. I also tried to give \n which is also giving me same error. `

data = ‘{“firstName”: “John”}\n
[here in postman \n is replaced by entering new line at end of json]’

`

vbrozik · September 21, 2022, 7:52am

It is difficult to help you without seeing what you are doing. If you want to continue, please show:

your Python code
the session in your console where you run the script, include the full output
the working request in Postman - I think it is best to open the Postman Console (in the lower left corner), expand the request, its headers, body and response

Please put the textual information between lines with triple backticks so we see it properly (your previous post shows invalid Python code). You use it like this:

```
Your text will be here.
```

Harsh · September 21, 2022, 11:57am

Python code.

URL = "http://xxx.xxxx.xxxxxx.xxx/elasticsearch/_msearch"

data = '{"index":"xxx-xxxxx-*","ignore_unavailable":true,"timeout":30000,"preference":1663049674576}
{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30s","time_zone":"Asia/Kolkata","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":[{"field":"@timestamp","format":"date_time"},{"field":"chunk.consumedAt","format":"date_time"},{"field":"consumedAt","format":"date_time"}],"query":{"bool":{"must":[{"query_string":{"query":"arId: \"xxxxx\"","analyze_wildcard":true,"default_field":"*"}},{"range":{"@timestamp":{"gte":1663049921550,"lte":1663050821550,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}\\n'

responsedata = requests.post(url=URL,auth=('xyz','xxxxxxxx'),data=data,headers = {"kbn-xsrf": "true", "Content-Type" : "application/json" })
print(responsedata.json())

Response of above given python request:

{'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'The msearch request must be terminated by a newline [\n]'}], 'type': 'illegal_argument_exception', 'reason': 'The msearch request must be terminated by a newline [\n]'}, 'status': 400}

Postman Details:

API : POST http://xxx.xxxxx.xxx.net/elasticsearch/_msearch
code : 200

Request Headers: 

kbn-xsrf: true
Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxx=
Content-Type: application/json
User-Agent: PostmanRuntime/7.29.2
Accept: */*
Cache-Control: no-cache
Postman-Token: xxxxxxxxxxxxxxxxxxxxxxxxxxx
Host: xxx.xxxx.xxxx.net
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 921
--------------------------------------------------------- 

Response : Headers
Server: nginx/1.10.3 (Ubuntu)
Date: Wed, 21 Sep 2022 09:33:48 GMT
Content-Type: application/json; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
kbn-name: kibana
kbn-xpack-sig: xxxxxxxxxxxxxxxxxxxxxxxxx
cache-control: no-cache
vary: accept-encoding
content-encoding: gzip

--------------------------------------------------------- 
Response Body : 
{
    "responses": [
        {
            "took": 277,
            "timed_out": false,
            "_shards": {
                "total": 70,
                "successful": 70,
                "skipped": 65,
                "failed": 0
            }
         }
    ]
}

Request Body:

Here you can see I have entered new line in the end of the body which works well in postman.

vbrozik · September 21, 2022, 8:25pm

I noticed few problems in your Python code.

The variable requests is undefined. I am guessing you accidentally deleted import requests during anonymizing the code.
You cannot put a literal newline in a normal string enclosed between single quotes '. I am guessing you accidentally inserted the newline during anonymizing the code.
At the end of the string you did not add a newline character. I tried to explain this to you. I will try again in different words below.
There is a related problem with your combination of characters \" in the JSON string.
The line with the JSON string is extremely long. Normally you should keep the length of the lines under 80 or under about 100 characters.

Backslash in a string

Backslash in a Python string is a special character. Some examples of characters you want to get and how to write the string (including quotes):

a newline character: '\n'
a backslash \: '\\'

Demonstration showing the resulting string in hex ASCII codes, please test it yourself and play with other variants:

>>> print(bytes('\\', 'ascii').hex())  # a single backslash
5c
>>> print(bytes('\n', 'ascii').hex())  # a newline character
0a
>>> print(bytes('\\n', 'ascii').hex()) # string of two characters: backslash and "n"
5c6e
>>> print(bytes('\"', 'ascii').hex())  # single character: double quotes
22
>>> print(bytes('\\"', 'ascii').hex()) # string of two characters: backslash and double quotes
5c22

So at the end of your string according to your description you want \n (not \\n). In the middle of the string to get a valid JSON you probably want \\" (not \").

How to write a long string over multiple lines

data = 'very long string'

Exactly the same string split over multiple lines of code:

data = (
    'very '
    'long '
    'string'
    )

ibra · October 14, 2022, 2:20pm

Any Idea how to solve that, having the same issue

Harsh · October 17, 2022, 7:48am

I’m still having the same issue. No idea how to solve it. I will share if I found a solution
Please share if you find any solution.

vbrozik · October 17, 2022, 8:54am

Did you apply my suggestions? Did you fix the problems I pointed out? If you need more explanation please be specific about what is difficult to understand in my suggestions. If we need to see your current code, please show it clean of syntactic errors - i.e. exactly the code which you are running without throwing exceptions.

Other approach could be to capture the actual communication using tcpdump or Wireshark. Then we will be able to see the difference between the working and not-working request. From the difference it should be easy to fix your failing request. Will you be able to do that?

Harsh · October 17, 2022, 9:16am

Yes I have tried all solutions you suggested but that thing did not work.
Ok sure I will try your suggestion & revert back to you.
Thanks.

Harsh · October 17, 2022, 1:25pm

  E..L.a@.@......[.......P..|..f.s.....d..........U=..POST./elasticsearch/_msearch.HTTP/1.1..kbn-xsrf:.true..Authorization:.Basic.XXXXXXXXXXXXXXXXXX=..Content-Type:.application/json..User-Agent:.PostmanRuntime/7.29.2..Accept:.*/*..Cache-Control:.no-cache..Postman-Token:.xxxxxxxxxxxxxxxxxxxxxxxxxxxxx..Host:.xxxxxxxxxx..Accept-Encoding:.gzip,.deflate,.br..Connection:.keep-alive..Content-Length:.921....{"index":"pm2-lspqs-*","ignore_unavailable":true,"timeout":30  000,"preference":1666002448430}.{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30s","time_zone":"Asia/Kolkata","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":[{"field":"@timestamp","format":"date_time"},{"field":"chunk.consumedAt","format":"date_time"},{"field":"consumedAt","format":"date_time"}],"query":{"bool":{"must":[{"query_string":{"query":"arId:.\"XXXXX\"","analyze _wildcard":true,"default_field":"*"}},{"range":{"@timestamp":{"gte":1666001570834,"lte":1666002470834,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2       147483647}}.

I have used tcpdump command & I’m Sharing you the details & contents which I found using it.
Above given packet content is for working request which I sent using postman. FYI I have added line break at the end of postman request body.

Harsh · October 17, 2022, 1:27pm

E..bz.@.@.+q...[.......P.....!y. .....>..........UF..POST./elasticsearch/_msearch.HTTP/1.1..Host:.xxxxxxxxxxxxxxx..User-Agent:.python-requests/2.28.1..Accept-Encoding:.gzip,.deflate..Accept:.*/*..Connection:.keep-alive..kbn-xsrf:.true..Content-Type:.application/json..Content-Length:.917..Authorization:.Basic.xxxxxxxxxxxxxxxxxx=..E...z.@.@.)....[.......P.....!y.....p...........UF..{"index":"pm2-lspqs-*","ignore_unavailable":true,"timeout":30000,"preference":1666002448430}{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":" @timestamp","interval":"30s","time_zone":"Asia/Kolkata","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":[{"field":"@timestamp","format":"date_time"},{"field":"chunk.consumedAt","format":"date_time"},{"field":"consumedAt","format":"date_time"}],"query":{"bool":{"must":[{"query_string":{"q uery":"arId:."xxxxx"","analyze_wildcard":true,"default_field":"*"}},{"range":{"@timestamp":{"gte ":1666001570834,"lte":1666002470834,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}

Now this one is for the python request in which im not able to add line break at the end of body. Which is also the reason why the request is failing. Please update me after checking the difference between working & non-working requests.

ibra · October 17, 2022, 1:35pm

setting header to headers={‘Content-Type’: ‘application/x-ndjson’} solved for me, or at least it is not this error anymore

Harsh · October 17, 2022, 1:40pm

Can you please share your request body here so I can get proper idea.
Thanks.

ibra · October 17, 2022, 1:49pm

here is my request body

b’{}\n {“id”: “my-index”, “params”: {“polygon”: [[ {pairs of GPS coordinate} ]]}}\n’

sorry i removed all real data for security reasons

ibra · October 17, 2022, 1:50pm

also it get request not post as per the documentation of Elasticsearch

vbrozik · October 17, 2022, 4:01pm

When you compare the body part of the captures you can notice some differences:

// capture 2
"timeout":30  000
//          ^--- Spaces inside a number? This makes the JSON invalid.

// capture 1
"q uery":"arId:."xxxxx""
//               ^--- You cannot put x or something else after closed double quotes.
//                    I notified you about this mistake of yours. You have to escape
//                    the double quotes if you need them as a characetr inside
//                    the JSON string. The string must contain a backslash: \"
//^--- What is this character? a space? I guess it is another mistake.

// capture 1
"analyze _wildcard":true
//      ^--- What is this character? a space?

// capture 1
"fragment_size":2       147483647}}.
//                                 ^--- What is this character? I think you let tcpdump
//                                      to replace unprintable characters with a dot.
//                                      Is it CR, LF or something else? Please confirm.
//               ^--- Spaces inside a number again? -> invalid JSON

Please see my post about escaping inside python strings again.

So I again see the mistake with the missed backslash. It seems you did not fix it: Instead of this sequence in the JSON string "query":"arId: \"xxxxx\"" you probably need: "query":"arId: \\"xxxxx\\""
We do not see what is the last character in the first capture because you let the the capture tool replace it by a dot. If it is LF, put \n inside your python string, if it is CR, put \r there. If you need CRLF, put \r\n there.

Suggestion: If you want to see what is the resulting string after the escape sequences are processed by Python, just print it:

>>> string = 'Lorem\nipsum\tdolor backslash quote: \" two backslashes quote: \\"'
>>> print(string)
Lorem
ipsum   dolor backslash quote: " two backslashes quote: \"

Please compare yourself the captures carefully for all the differences. To see the individual bytes (for example to confirm the last character) use the option -X of tcpdump.

Harsh · October 18, 2022, 4:19am

The dot that you are asking about is newline that I inserted using Enter key after the json ended. tcpdump replaced it by dot. FYI I am already using -X in tcp command.

Topic		Replies	Views
Python updated json file is full of backslashes and newline characters Python Help	9	12556	January 24, 2023
Python Replace String in json output Python Help help	1	14529	February 26, 2022
Specifying curl command in a python list of strings Python Help	1	738	January 6, 2023
Use newlines \n with the command python -c Python Help help	3	3480	October 18, 2022
Network path format Python Help	12	1218	December 5, 2023

How can I add \n(newline) In Json string

Backslash in a string

How to write a long string over multiple lines

Related Topics