# Download several files from github

**URL:** <https://discuss.python.org/t/download-several-files-from-github/9156>\
**Category:** Python Help\
**Tags:** git\
**Created:** [June 7, 2021, 9:20pm UTC](https://discuss.python.org/t/download-several-files-from-github/9156 "2021-06-07T21:20:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mfernandes](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@mfernandes](https://discuss.python.org/u/mfernandes)\
**Post date:** [June 7, 2021, 9:20pm UTC](https://discuss.python.org/t/download-several-files-from-github/9156/1 "2021-06-07T21:20:24Z")

</div>

Dear Python community,  
I am brand new to Python.  
I would like to download all the files available on this Github link: [professions-foi-candidats/documents/LG17/1 at master · regardscitoyens/professions-foi-candidats · GitHub](https://github.com/regardscitoyens/professions-foi-candidats/tree/master/documents/LG17/1) .  
Is it possible through python to do the download of all files and save them in my laptop?  
From what I understood, if I want to download a file I should code:

```auto
import urllib
urllib.urlretrieve ("https://github.com/regardscitoyens/professions-foi-candidats/blob/master/documents/LG17/1/LG17-1-1-BLATRIX-CONTAT-2-tour1-profession_foi.pdf", "mydirectory/myfile.pdf")

```

the issue in this case is that the files are in different links.

---

<div class="post-metadata">

**Author:** ![cameron](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cameron/32/2658_2.png) [@cameron](https://discuss.python.org/u/cameron)\
**Post date:** [June 7, 2021, 11:12pm UTC](https://discuss.python.org/t/download-several-files-from-github/9156/2 "2021-06-07T23:12:26Z")

</div>

You could use beautifulsoup to fetch the page with the listing, grab all  
the URLs from the contents, then fecth each as above.

The bs4 library:

```
https://pypi.org/project/beautifulsoup4/

```

Cheers,  
Cameron Simpson [cs@cskk.id.au](mailto:cs@cskk.id.au)

---

<div class="post-metadata">

**Author:** ![Mariatta](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mariatta/32/30581_2.png) [@Mariatta](https://discuss.python.org/u/Mariatta)\
**Post date:** [June 7, 2021, 11:33pm UTC](https://discuss.python.org/t/download-several-files-from-github/9156/3 "2021-06-07T23:33:39Z")

</div>

Is there a reason you’re not doing `git checkout ` to get the files from GitHub to your computer?

`https://github.com/regardscitoyens/professions-foi-candidats.git`

---

<div class="post-metadata">

**Author:** ![erlendaasland](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/erlendaasland/32/4378_2.png) [@erlendaasland](https://discuss.python.org/u/erlendaasland)\
**Post date:** [June 8, 2021, 10:31am UTC](https://discuss.python.org/t/download-several-files-from-github/9156/4 "2021-06-08T10:31:36Z")

</div>

Sounds like a good case for trying out [git sparse checkout](https://www.git-scm.com/docs/git-sparse-checkout).

---

<div class="post-metadata">

**Author:** ![mfernandes](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@mfernandes](https://discuss.python.org/u/mfernandes)\
**Post date:** [June 8, 2021, 11:00am UTC](https://discuss.python.org/t/download-several-files-from-github/9156/5 "2021-06-08T11:00:15Z")

</div>

Thank you for your comments.  
I tried to understand Cameron Simpson advice on using beautifulsoup. I coded:  
html\_doc =copy&paste their html which was long  
from bs4 import BeautifulSoup  
soup = BeautifulSoup(html\_doc, ‘html.parser’)

print(soup.prettify())

and I received the error:  
“File “”, line 104  
  
^  
IndentationError: unindent does not match any outer indentation level”

So I think I will try to understand how git sparse checkout works.  
Thank you!

---

<div class="post-metadata">

**Author:** ![cameron](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cameron/32/2658_2.png) [@cameron](https://discuss.python.org/u/cameron)\
**Post date:** [June 8, 2021, 11:18pm UTC](https://discuss.python.org/t/download-several-files-from-github/9156/6 "2021-06-08T23:18:36Z")

</div>

That looks to me as though you pasted HTML straight into your Python  
code. And Python then tries to parse the HTML as Python. Badness.

The cleanest way to try this is to put the HTML into a separate file, eg  
named “foo.html”. Then:

```
from bs4 import BeautifulSoup
with open("foo.html") as f:
    html_doc = f.read()
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.prettify())

```

which keeps the HTML code separate from the Python code.

Cheers,  
Cameron Simpson [cs@cskk.id.au](mailto:cs@cskk.id.au)
