# Python: Extract Data from an Interactive Map on the Web, with Several Years, into a CSV file **URL:** \ **Category:** Python Help\ **Created:** [April 4, 2024, 12:41pm UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266 "2024-04-04T12:41:46Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 4, 2024, 12:41pm UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/1 "2024-04-04T12:41:46Z")
Hi Python Community, I hope you are doing well. I have a quite challenging task to do (or at least for me, as I am novice in Python). I need some help to extract huge amounts of data from an interactive map on a website and put them into a `.csv` file, and would like to know if there is a way to extract all the data from it. The map in question is [here](https://www.cohispania.com/wp-content/uploads/2024/01/mapa-2023.html). 1. I would like to know if there is a way to extract all the municipalities’ data, please? I thank you in advance for your help. Best wishes, Michael **Edit:** Here is what I tried, but in vein, to extract those data into Python data frame. I know that the data that I want are there: ```python { "type": "MultiPolygon", "arcs": [ [ [ -15410, 15526, -15491, -15236, -15379 ] ] ], "properties": { "NAMEUNIT": "Municipio: Villabrázaro\u003c/strong>
", "Unitario": "Precio unitario medio : 564 Euros/m2\u003c/sup>
", "Precio unitario medio del municipio": 564, "desviacion": "Desviacion tipica: 369
", "superficie": "Superficie media: 311 m2\u003c/sup>
", "moda": "Rango de precio mas frecuente: 200-400 Euros/m2\u003c/sup>
", "poblacion": "Poblacion: 239
", "renta_persona": "Renta media por persona: 13487 Euros/año
", "renta_hogar": "Renta media por hogar : 31180 Euros/año
" } }, { "type": "MultiPolygon", "arcs": [ [ [ -15348, -15345, -15267, -13840, -13292 ] ] ], "properties": { "NAMEUNIT": "Municipio: Villaescusa\u003c/strong>
", "Unitario": "Precio unitario medio : 580 Euros/m2\u003c/sup>
", "Precio unitario medio del municipio": 580, "desviacion": "Desviacion tipica: 660
", "superficie": "Superficie media: 242 m2\u003c/sup>
", "moda": "Rango de precio mas frecuente: 100-200 Euros/m2\u003c/sup>
", "poblacion": "Poblacion: 235
", "renta_persona": "Renta media por persona: 11123 Euros/año
", "renta_hogar": "Renta media por hogar : 21876 Euros/año
" } }, and so on... ``` I tried the following chunk, but obtained an error that I cannot solve: ```python import requests import pandas as pd # URL of the web page containing the data url = "https://www.cohispania.com/wp-content/uploads/2024/01/mapa-2023.html" # Fetch the HTML content response = requests.get(url) # Check if the request was successful if response.status_code == 200: # Extract JSON data from HTML content json_data_list = response.json() # List to store extracted data for each municipality extracted_data = [] # Iterate over each JSON object representing a municipality for json_data in json_data_list: # Extract relevant information from the JSON data properties = json_data["properties"] name = properties["NAMEUNIT"].split(":")[1].strip().replace("
", "") precio_unitario_medio = properties["Precio unitario medio del municipio"] desviacion = properties["desviacion"].split(":")[1].strip().replace("
", "") superficie = properties["superficie"].split(":")[1].strip().replace(" m2
", "") moda = properties["moda"].split(":")[1].strip().replace("
", "") poblacion = properties["poblacion"].split(":")[1].strip().replace("
", "") renta_persona = properties["renta_persona"].split(":")[1].strip().replace(" Euros/año
", "") renta_hogar = properties["renta_hogar"].split(":")[1].strip().replace(" Euros/año
", "") # Append extracted data to the list extracted_data.append({ "Name": name, "Precio unitario medio": precio_unitario_medio, "Desviacion": desviacion, "Superficie": superficie, "Moda": moda, "Poblacion": poblacion, "Renta media por persona": renta_persona, "Renta media por hogar": renta_hogar }) # Create DataFrame from the extracted data df = pd.DataFrame(extracted_data) # Display the DataFrame print(df) else: print("Failed to fetch data from the web page") ``` - Could anyone please give me a hand with this? ---
**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\ **Post date:** [April 5, 2024, 7:58am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/2 "2024-04-05T07:58:44Z")
> [@michaeldg\_94](#): > > I tried the following chunk, but obtained an error that I cannot solve: > > `` 1. What error did you get? 2. For the example input that you show, exactly what result do you want to get? What are the rules that the code should use, to calculate that? ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 8:05am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/3 "2024-04-05T08:05:55Z")
Hi @kknechtel , Thank you for reaching out. Below are the answers to your points. Thank you for your willingness to help: 1. Here are the errors obtained: ```python --------------------------------------------------------------------------- JSONDecodeError Traceback (most recent call last) File ~\AppData\Local\anaconda3\Lib\site-packages\requests\models.py:971, in Response.json(self, **kwargs) 970 try: --> 971 return complexjson.loads(self.text, **kwargs) 972 except JSONDecodeError as e: 973 # Catch JSON-related errors and raise as requests.JSONDecodeError 974 # This aliases json.JSONDecodeError and simplejson.JSONDecodeError File ~\AppData\Local\anaconda3\Lib\json\ __init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 343 if (cls is None and object_hook is None and 344 parse_int is None and parse_float is None and 345 parse_constant is None and object_pairs_hook is None and not kw): --> 346 return _default_decoder.decode(s) 347 if cls is None: File ~\AppData\Local\anaconda3\Lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w) 333 """Return the Python representation of ``s`` (a ``str`` instance 334 containing a JSON document). 335 336 """ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end() File ~\AppData\Local\anaconda3\Lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx) 354 except StopIteration as err: --> 355 raise JSONDecodeError("Expecting value", s, err.value) from None 356 return obj, end JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: JSONDecodeError Traceback (most recent call last) Cell In[17], line 13 10 # Check if the request was successful 11 if response.status_code == 200: 12 # Extract JSON data from HTML content ---> 13 json_data_list = response.json() 15 # List to store extracted data for each municipality 16 extracted_data = [] File ~\AppData\Local\anaconda3\Lib\site-packages\requests\models.py:975, in Response.json(self, **kwargs) 971 return complexjson.loads(self.text, **kwargs) 972 except JSONDecodeError as e: 973 # Catch JSON-related errors and raise as requests.JSONDecodeError 974 # This aliases json.JSONDecodeError and simplejson.JSONDecodeError --> 975 raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) JSONDecodeError: Expecting value: line 1 column 1 (char 0) ``` 1. I would like to obtain a data frame in Python, in which each column represents one variable: - (so `NAMEUNIT` for the first column, `Unitario` for the second one, and so on). - For each row, I would like to extract the data from it: `Villabrázaro` for the first column, `564 Euros/m^2` for the second one, and so on, for all the municipalities available on the HTML. I apologize, it is the first time I am trying to extract data from a map, so maybe I am not clear. Thank you for your help! ---
**Author:** ![FelixLeg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/felixleg/32/18439_2.png) [@FelixLeg](https://discuss.python.org/u/FelixLeg)\ **Post date:** [April 5, 2024, 9:26am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/4 "2024-04-05T09:26:26Z")
> [@michaeldg\_94](#): > > `json_data_list = response.json()` Are you sure the response contains a _valid_ JSON text? Maybe the service responds with a _HTML_ content (because there was an error)? ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 9:30am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/5 "2024-04-05T09:30:05Z")
Hi @FelixLeg, Thank you for reaching out. You are true: I am not sure about the validity of the JSON. To be honest, I don’t know how I can really know that it. Sorry, I am still novice. The data that I want is always in this form, with changing names and values: ```python "geometries": [ { "type": "MultiPolygon", "arcs": [ [ [ 0, 1, 2 ] ] ], "properties": { "NAMEUNIT": "Municipio: Mélida\u003c/strong>
", "Unitario": "Precio unitario medio : 729 Euros/m2\u003c/sup>
", "Precio unitario medio del municipio": 729, "desviacion": "Desviacion tipica: 398
", "superficie": "Superficie media: 190 m2\u003c/sup>
", "moda": "Rango de precio mas frecuente: 200-400 Euros/m2\u003c/sup>
", "poblacion": "Poblacion: 714
", "renta_persona": "Renta media por persona: 14526 Euros/año
", "renta_hogar": "Renta media por hogar : 34162 Euros/año
" }}, ``` Thank you for your help! ---
**Author:** ![FelixLeg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/felixleg/32/18439_2.png) [@FelixLeg](https://discuss.python.org/u/FelixLeg)\ **Post date:** [April 5, 2024, 9:35am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/6 "2024-04-05T09:35:17Z")
Well, I think the only way to test if this is a JSON is to call `response.json()` and catch any `Exception`s with `try: ... except:...` construction 🙂 ~~Anyway, try to load the “JSON” data manually (using your browser or a tool like `wget` or `curl`) and check the contents~~ From what you posted it is _almost_ a JSON, but with errors. Your data has one issue: it is a _list_ of JSONs not one entry. They always do return JSONs like this? ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 9:39am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/7 "2024-04-05T09:39:07Z")
> [@FelixLeg](#): > > They always do return JSONs like this? Yes, the data are always in this form, **except** for municipalities with missing data: ```python [ "Municipio: Valderrebollo<\/strong>
Precio unitario medio: No disponible
Desviacion tipica: No disponible
Superficie media: No disponible
Rango de precio mas frecuente: No disponible
Poblacion: 23
Renta media por persona: 14898 Euros/año
Renta media por hogar: 25639 Euros/año", "Municipio: Mirafuentes<\/strong>
Precio unitario medio: No disponible
Desviacion tipica: No disponible
Superficie media: No disponible
Rango de precio mas frecuente: No disponible
Poblacion: 59
Renta media por persona: 15444 Euros/año
Renta media por hogar: 38614 Euros/año", "Municipio: Valdesotos<\/strong>
Precio unitario medio: No disponible
Desviacion tipica: No disponible
Superficie media: No disponible
Rango de precio mas frecuente: No disponible
Poblacion: 27
Renta media por persona: 14898 Euros/año
Renta media por hogar: 25639 Euros/año", ... ], ``` ---
**Author:** ![FelixLeg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/felixleg/32/18439_2.png) [@FelixLeg](https://discuss.python.org/u/FelixLeg)\ **Post date:** [April 5, 2024, 9:40am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/8 "2024-04-05T09:40:19Z")
Can you give me an example URL of that service? ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 9:42am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/9 "2024-04-05T09:42:33Z")
> Can you give me an example URL of that service? What do you mean? The full map is here: [leaflet (cohispania.com)](https://www.cohispania.com/wp-content/uploads/2024/01/mapa-2023.html) ---
**Author:** ![FelixLeg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/felixleg/32/18439_2.png) [@FelixLeg](https://discuss.python.org/u/FelixLeg)\ **Post date:** [April 5, 2024, 9:44am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/10 "2024-04-05T09:44:51Z")
Sorry I’s typing too fast 😊 I wanted the URL from what you get the _JSON_ data. I want to check the data on my own. ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 9:56am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/11 "2024-04-05T09:56:02Z")
Ok, I understand. Basically, the Leaflet map is located in the URL file provided in #9. Then I just type `Ctrl + U` in Windows (or in Mac format `Option + Command + U` if not mistaken) to have the full source page. Thank you in advance for your help! ---
**Author:** ![FelixLeg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/felixleg/32/18439_2.png) [@FelixLeg](https://discuss.python.org/u/FelixLeg)\ **Post date:** [April 5, 2024, 10:06am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/12 "2024-04-05T10:06:59Z")
O…M…G… the page is so big that none of my editors wants to load them 😲 I think that you request from the `response.json()` too much. If the data is _embedded_ inside a HTML file then `json()` doesn’t know how to parse it. It is not a magical tool, it can’t extract JSON on its own. I think you have to write ~~some~~ a lot of code that extracts the JSON first on your own. ☹ ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 10:38am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/13 "2024-04-05T10:38:43Z")
Hi again Przemysław, - So that’s why `response.json()` was giving me an error: > The data is _embedded_ inside a HTML file then `json()` doesn’t know how to parse it - Their code is really vague. Could this be a strategy to dissuade people from owning their data? - Do you think extraction is possible still, please? Thanks again for your help, time and generosity @FelixLeg! ---
**Author:** ![FelixLeg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/felixleg/32/18439_2.png) [@FelixLeg](https://discuss.python.org/u/FelixLeg)\ **Post date:** [April 5, 2024, 10:53am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/14 "2024-04-05T10:53:06Z")
I wish you luck, however from what I’s able to get from the site source, it may be hard. Whoever wrote the source of the site he/she has made _everything_ to make it impossible to extract any data ☹ The code is obfuscated and very big. ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 11:03am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/15 "2024-04-05T11:03:59Z")
@FelixLeg: Thank you again for your amazing help. Probably. It is a consultancy firm in Spain, so I imagine that they made it difficult to extract data on purpose… I wish you all the best. Again, thank you! ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 5, 2024, 12:41pm UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/16 "2024-04-05T12:41:37Z")
If anyone has a way around this problem, I’m more than happy to “hear” it. Thank you! ---
**Author:** ![FelixLeg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/felixleg/32/18439_2.png) [@FelixLeg](https://discuss.python.org/u/FelixLeg)\ **Post date:** [April 5, 2024, 4:19pm UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/17 "2024-04-05T16:19:21Z")
Okay, I found a way to extract JSON from the site 😃 ```python import urllib.request as req import html.parser as hp resp = req.urlopen("https://www.cohispania.com/wp-content/uploads/2024/01/mapa-2023.html") html_cont = resp.read().decode('utf8') class html_parser(hp.HTMLParser): def __init__ (self): super(). __init__ () self.inside_data = False self.the_data = "" def handle_starttag(self, tag, attrs): if not self.inside_data: if tag == 'script': for k,v in attrs: if k == 'type' and v == 'application/json': self.inside_data = True return def handle_endtag(self, tag): if self.inside_data: if tag == 'script': self.inside_data = False def handle_data(self, data): if self.inside_data: self.the_data += data parser = html_parser() parser.feed(html_cont) the_json = parser.the_data parser.close() #optional: write to file with open('result.json', 'w') as f: f.write(the_json) ``` Now it is up to you what you want to do it with the data 😉 ---
**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\ **Post date:** [April 5, 2024, 10:00pm UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/18 "2024-04-05T22:00:25Z")
> [@michaeldg\_94](#): > > ```python > # Extract JSON data from HTML content > json_data_list = response.json() > > ``` This is completely wrong. First off, `.json()` doesn’t do that - it expects **the entire data** from the request to be a single chunk of JSON data. You can’t use it for a web page because the data is in a completely different format. Second, the HTML content in this case **doesn’t contain** the JSON you want anyway. Instead, it contains JavaScript code that your browser uses to make AJAX calls to get the JSON data. Please read: > ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 8, 2024, 5:39am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/19 "2024-04-08T05:39:45Z")
Hi @FelixLeg, I just want to thank you so much for your help in this! It works perfectly well. I have learned a lot with you in this thread! Thanks for sharing your knowledge. Hope to “hear” from you again in one of mine future posts. All the best, Michael ---
**Author:** ![michaeldg\_94](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@michaeldg\_94](https://discuss.python.org/u/michaeldg_94)\ **Post date:** [April 8, 2024, 5:42am UTC](https://discuss.python.org/t/python-extract-data-from-an-interactive-map-on-the-web-with-several-years-into-a-csv-file/50266/20 "2024-04-08T05:42:21Z")
Hi @kknechtel, Thank you for your clarifications about what I can do or mot with HTML and JavaScript. Sorry for my mistakes, I thought it was possible the way I presented it. I also learned more with you! Thank you very much to help me to become a little bit more knowledgeable on HTML. Again, thank you so much. All the best,