-
-
Notifications
You must be signed in to change notification settings - Fork 204
-
-
Notifications
You must be signed in to change notification settings - Fork 204
Error uploading large files #750
Comments
@WoLfulus @wellingguzman — is this an API, Docker, or server issue? |
@benhaynes I can't tell where's the issue. Hey @shartley76 can you tell us if there's any logs in the api |
hey @shartley76, can you go to the network tab, and find the failed request (the one with status 500) and screenshot or copy paste the response? Also look in the |
@shartley76 Sorry I didn't explain, that's a log from the application and I can't tell what really happening on the server side that throws these errors. I need to know what on the |
hi @wellingguzman, on network tab I don't have a response, just an internal server error 500 listed on request. Response tab is empty, no data. |
The request doesn't seem to be getting to the API. Is there any errors in your nginx logs? |
I can't see anything in the logs, they're empty |
for accessing the files resource there are 2 errors: the first is: I'm not sure why i'm getting that error when I can successfully upload to files a smaller file? error no2 is: |
also @wellingguzman checking the network timing on the files (server error 500 resp). I can see that it does start to receive a response: |
Hi @wellingguzman were you able to upload a large file 100Mb plus with a docker deployment, or stand alone? If so would you mind sharing your dockerfile and/or config settings? Then I can just verify I have everything setup correctly. Thanks |
Hey @shartley76, it was hard for me to know what was the error without seeing the logs in the API. I created a dummy 100mb file, and I am now getting an error on bigger files. I don't know why I wasn't before. The issue is the code trying to verify that a huge string is not an url. I am working on a fix right now. We would need to replace this function: https://github.com/directus/api/blob/master/src/helpers/file.php#L299 in something that doesn't need to read the whole thing or doesn't break trying to. Either way From Docs: http://php.net/manual/en/filter.filters.validate.php
|
Hey @shartley76, I believe I have fixed what was triggering the 500 internal error. In my end I am not experience this issue anymore on big files. I based the validation using regex, instead of Can you confirm it's working for you on Ref: 81582e1 |
Hey @wellingguzman I tried with that patch and I seem to still be getting a 500 when uploading this test file: http://ipv4.download.thinkbroadband.com/200MB.zip. |
@computerwizjared can you show me the full log? I am going to try using the 200MB and see if I can reproduce it. |
@computerwizjared Did you try upload the file via url? I found an issue uploading non-image files using url. Uploading files using url only works with image at the moment. That error message should be more friendly. |
@wellingguzman I am uploading the file using the built-in Directus App method... not sure if that is via url or not. And after switching to the Log Message: Maybe this is a different issue pertaining to my configuration? It's odd that I'm trying to upload a 200MB file and it is giving me an out of memory error. I feel like it shouldn't take more memory than the size of the file, but I may be wrong. Thank you! |
I am going to properly fix this bug, I didn't though it was going to be a problem, but I am going to reduce the passing around the file content and encoding decoding the base64 content. It's not related to the configuration but the API handling big files poorly, keeping these huge data on memory instead of saving into disc as soon as possible to avoid the high memory usage. I will let you know as soon as I got it running as expected. Thanks for log. |
@wellingguzman are you able to push any changes for the api to the directus/api docker registry? I'm running docker (linux containers) on windows locally. Many thanks! |
@shartley76 I couldn't push any changes, as I am not working with the docker image. @WoLfulus may be able to help you. I will be working on a proper fix to avoid huge files crashing the server. |
Hey @shartley76 I ran into trouble trying to test Directus changes with Docker and the best way I found was to just grab the modified files and use the Dockerfile |
thanks @computerwizjared, i'll grab latest from master and retry as you say. |
Hi @computerwizjared I tried running the latest api code from master on a docker container with 1gb memory. I'm trying to upload 144Mb video file, but still getting an error on trying to upload. upload_max_filesize=500M Is there something i'm missing in terms of config? |
@shartley76 you're right, I'm getting a 502 gateway timeout. Upping the memory limit helped with the one error I was getting before (with a slightly smaller file) but now it's a different one. |
@computerwizjared do you think that this bug will be able to get assigned to someone soon? It has been in the high priority bug triage for a while now and is a blocker for us at the moment |
@shartley76 I don't have any idea. I'm not affiliated with Directus, just another user. We're having this issue too, and all we're doing right now is uploading to the server directly and then editing the database manually. Sorry! |
@shartley76 and @computerwizjared — we'll try to get this resolved asap, no specific timing. The more info we have the better/faster. @hemratna + @theharshin — maybe we can look into this one? |
Sure @benhaynes. Will look into the issue this weekend. Thanks for the inputs @shartley76 @computerwizjared 🙂 |
* Return object in delete after hook instead of onli ID (#882) * Add fix for big file sizes Closes #750 * Add migrations for hash and single-file * Show correct fields in roles.users For some reason the database column for options was empty Closes https://github.com/directus/app/issues/1471 * Delete ISSUE_TEMPLATE.md * security notice * Add check for mod_php before setting php_value for upload size This will prevent errors on systems that don't allow overriding the php values from within the .htaccess files. This will only check for php 7+ though, as the mod_php directive is version specific. This is okay for now, as we officially only support PHP 7.1+ * Change field width from integer to string This will allow the app to render the fields in the correct widths starting with v7.2. * Issue fix #854 (#896) * Add migrations for setting field notes and widths Lays out the settings a bit nicer and adds setting descriptions. Fixes https://github.com/directus/app/issues/1379 * Fix sort order of fields on install * Increase specificity of migrations so it doesn't target non-settings * Move collection notes to the DB I'm aware that this makes them english only for the time being. Once we implement the using the translation column in the app, we can make them properly translatable. * Fix abstraction name * Add migrations for misc fields Sorting of files, making a couple interfaces required, etc * Bump version * Fix: Wrong MIME for extentions in uppercase (#895) * FEAT more events that invalidate the cache (#892) * Allowing string relations (#800) * emoji support for comments and bookmark names * Use JSON interface for system collections * Fixing custom primary key primary key column name (#881) Swapping this variable seems to resolve the issue. * Bump version * Issue #885 (#898) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * Issue#885 - Done * #885 Removed Test cases * Issue #886 (#899) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * Issue#885 - Done * Issue#886 - Done * #886 Reverted unwanted code * Issue #884 (#901) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #884 Done * #884 Removed Test cases * Issue #884 - Change (#907) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #884 Done * #884 Removed Test cases * #884 change * Fix#810 (#908) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #810 done * #810 Reverting Test Cases * Issue Fix #902 (#909) * Issue fix #902 * Add migration for allow value nullable in settings table * Set texttype for value field * Doc issue fix #84 (#910) * Issue fix #841 (#911) * Increase expiry time of tokens from 5 to 20 minutes (#913) It should still be pretty secure. This allows the app to go easier on the refreshing, and it makes sure that you can upload large files without having the token expire halfway through. * Fix missing ref to 5 min exp * Issue Fix #863 (#916) * Issue fix #853 (#918) * Issue Fix #920 (#922) * Issue Fix #920 * Issue Fix #920 * Generate GraphQL schema file which support primary-key, text-input and numeric interfaces. * Implement graphql-php server. * GraphQL type for DirectUs files * Code cleanup. * Adding custom scalar support for Date, Datetime, JSON * Adding support for the m2o type in schema generation. * Adding support for m2o type. * Support for O2M. GraphQL type for Directus Role. Rename GraphQL types for Directus user, files. * Adding pagination support. * Code cleanup. * Adding time scalar type. * Adding meta support. * Search filter approach 1. * Search result. * Adding support for AND and OR logical filter. * Rebase with master. * Adding support for Activity, Collection Preset, Collection, Field type. Change the naming convention. Adding pascal case function in string utils. * Adding support for setting collection. * Adding support for Folder, Permission, Relation, Revision collection. * Adding README.md * Update README.md * Change naming convention to snake_case.
* Return object in delete after hook instead of onli ID (#882) * Add fix for big file sizes Closes #750 * Add migrations for hash and single-file * Show correct fields in roles.users For some reason the database column for options was empty Closes https://github.com/directus/app/issues/1471 * Delete ISSUE_TEMPLATE.md * security notice * Add check for mod_php before setting php_value for upload size This will prevent errors on systems that don't allow overriding the php values from within the .htaccess files. This will only check for php 7+ though, as the mod_php directive is version specific. This is okay for now, as we officially only support PHP 7.1+ * Change field width from integer to string This will allow the app to render the fields in the correct widths starting with v7.2. * Issue fix #854 (#896) * Add migrations for setting field notes and widths Lays out the settings a bit nicer and adds setting descriptions. Fixes https://github.com/directus/app/issues/1379 * Fix sort order of fields on install * Increase specificity of migrations so it doesn't target non-settings * Move collection notes to the DB I'm aware that this makes them english only for the time being. Once we implement the using the translation column in the app, we can make them properly translatable. * Fix abstraction name * Add migrations for misc fields Sorting of files, making a couple interfaces required, etc * Bump version * Fix: Wrong MIME for extentions in uppercase (#895) * FEAT more events that invalidate the cache (#892) * Allowing string relations (#800) * emoji support for comments and bookmark names * Use JSON interface for system collections * Fixing custom primary key primary key column name (#881) Swapping this variable seems to resolve the issue. * Bump version * Issue #885 (#898) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * Issue#885 - Done * #885 Removed Test cases * Issue #886 (#899) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * Issue#885 - Done * Issue#886 - Done * #886 Reverted unwanted code * Issue #884 (#901) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #884 Done * #884 Removed Test cases * Issue #884 - Change (#907) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #884 Done * #884 Removed Test cases * #884 change * Fix#810 (#908) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #810 done * #810 Reverting Test Cases * Issue Fix #902 (#909) * Issue fix #902 * Add migration for allow value nullable in settings table * Set texttype for value field * Doc issue fix #84 (#910) * Issue fix #841 (#911) * Increase expiry time of tokens from 5 to 20 minutes (#913) It should still be pretty secure. This allows the app to go easier on the refreshing, and it makes sure that you can upload large files without having the token expire halfway through. * Fix missing ref to 5 min exp * Issue Fix #863 (#916) * Issue fix #853 (#918) * Issue Fix #920 (#922) * Issue Fix #920 * Issue Fix #920 * Issue fix #879 (#924) * [thumbnailer] Support for files in subdirectories (#856) Many websites store images in a complex directory structure. This PR makes it possible to use thumbnailer in such cases. For instance : `/thumbnail/_/100/100/crop/good/complex/path/to/some-image.jpg * defaults cors.max-age to 600 (#921) * Bump version * Generate GraphQL schema file which support primary-key, text-input and numeric interfaces. * Implement graphql-php server. * GraphQL type for DirectUs files * Code cleanup. * Adding custom scalar support for Date, Datetime, JSON * Adding support for the m2o type in schema generation. * Adding support for m2o type. * Support for O2M. GraphQL type for Directus Role. Rename GraphQL types for Directus user, files. * Adding pagination support. * Code cleanup. * Adding time scalar type. * Adding meta support. * Search filter approach 1. * Search result. * Adding support for AND and OR logical filter. * Rebase with master. * Adding support for Activity, Collection Preset, Collection, Field type. Change the naming convention. Adding pascal case function in string utils. * Adding support for setting collection. * Adding support for Folder, Permission, Relation, Revision collection. * Adding README.md * Update README.md * Change naming convention to snake_case. * Change in naming convention. Merge list and single query into list query by adding additional arg `id`.
* pgsql 10 initial support * email_notification column must be set as a boolean * Handle unique column collisions * BUG delta in revisions can be null * BUG transformed the remaining lastInsertValue into getLastGeneratedId() * Pass new item flag to o2m new items Closes https://github.com/directus/app/issues/1418 * Don't show popover for 0 items / no template Closes https://github.com/directus/app/issues/1397 * Bug fix (#848) * Merge conflict resolve * Handle item not found exception in collection detail API * Extended the list of safe tags (#849) As described in issue #832 * Issue fix #819 (#851) * Mark adding new item as new in m2m * Bump version * Revert composer changes * Issue fix #843 (#852) * BUG searches with LIKE on non-textual columns * Remove the extensions from the API * Issue fix #847 (#857) * Issue fix #833 (#859) * Initial commit for documentation (#844) * Revert "Initial commit for documentation (#844)" (#868) This reverts commit 6e85d59. * BUG Bypass Zend-db choice not to allow nullable boolean fields * BUG field length were not taken into account * CHORE dupliacted line * BUG o2m working + post-alter table event dispatching * Return object in delete after hook instead of onli ID (#882) * Add fix for big file sizes Closes #750 * Add migrations for hash and single-file * Show correct fields in roles.users For some reason the database column for options was empty Closes https://github.com/directus/app/issues/1471 * Delete ISSUE_TEMPLATE.md * security notice * Add check for mod_php before setting php_value for upload size This will prevent errors on systems that don't allow overriding the php values from within the .htaccess files. This will only check for php 7+ though, as the mod_php directive is version specific. This is okay for now, as we officially only support PHP 7.1+ * Change field width from integer to string This will allow the app to render the fields in the correct widths starting with v7.2. * Issue fix #854 (#896) * Add migrations for setting field notes and widths Lays out the settings a bit nicer and adds setting descriptions. Fixes https://github.com/directus/app/issues/1379 * Fix sort order of fields on install * Increase specificity of migrations so it doesn't target non-settings * Move collection notes to the DB I'm aware that this makes them english only for the time being. Once we implement the using the translation column in the app, we can make them properly translatable. * Fix abstraction name * Add migrations for misc fields Sorting of files, making a couple interfaces required, etc * Bump version * Fix: Wrong MIME for extentions in uppercase (#895) * FEAT more events that invalidate the cache (#892) * Allowing string relations (#800) * emoji support for comments and bookmark names * Use JSON interface for system collections * Fixing custom primary key primary key column name (#881) Swapping this variable seems to resolve the issue. * Bump version * Issue #885 (#898) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * Issue#885 - Done * #885 Removed Test cases * Issue #886 (#899) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * Issue#885 - Done * Issue#886 - Done * #886 Reverted unwanted code * Issue #884 (#901) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #884 Done * #884 Removed Test cases * Issue #884 - Change (#907) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #884 Done * #884 Removed Test cases * #884 change * Fix#810 (#908) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #810 done * #810 Reverting Test Cases * Issue Fix #902 (#909) * Issue fix #902 * Add migration for allow value nullable in settings table * Set texttype for value field * Doc issue fix #84 (#910) * Issue fix #841 (#911) * Increase expiry time of tokens from 5 to 20 minutes (#913) It should still be pretty secure. This allows the app to go easier on the refreshing, and it makes sure that you can upload large files without having the token expire halfway through. * Fix missing ref to 5 min exp * Issue Fix #863 (#916) * Issue fix #853 (#918) * Issue Fix #920 (#922) * Issue Fix #920 * Issue Fix #920 * Issue fix #879 (#924) * [thumbnailer] Support for files in subdirectories (#856) Many websites store images in a complex directory structure. This PR makes it possible to use thumbnailer in such cases. For instance : `/thumbnail/_/100/100/crop/good/complex/path/to/some-image.jpg * defaults cors.max-age to 600 (#921) * Bump version * Fix 943 (#947) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #943 * Fix 717 (#944) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #717 * Fix 576 (#926) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #576 - In progress * #576 O2M and M20 nested filters * #576 Fix O2M and M2O nested filters * get proper string length (#933) Not tested... I only based this PR on: Ref: 0fce6a4#commitcomment-33408113 * fixed settings logo (#940) * added collection/table to InvalidFieldException (#956) * Fix 931 (#936) * Test cases : AUthentication - Auth, Forgot Password, Collections - Create, Delete * #931 * #931 * Issue fix #917 (#960) * reuse item service instead of using a new instance (#959) * Issue fix 762 (#961) * Plain text mail issu resolve (#966) * Bump version
Still getting this error with api version |
I could "fix" it for now by increasing the settings like suggested by @shartley76 and it works now for somewhat bigger files. But it would be amazing if there is a more stable version. So I ask myself it this issue still exists?
|
I believe that this issue is caused by the extra "processing" being done using up more memory than the file itself. Ideally Directus should allow for uploads up to the php.ini settings, but clearly that's not the case and users need to "pad" them higher for larger files. @bjgajjar @hemratna — any thoughts on where Directus might be eating up more memory or why we need to increase the values so much higher that the actual uploaded file size? Can we at least isolate which of these settings is the important one? |
hi @benhaynes are there any updates on this? Should i re-test with the latest api 2.0.21? Looks like there are still some issues as per your last post on this? Currently with the above settings the video file limit seems to be about 110mb. I can re-test with v2.0.21 to see if this has now been resolved? |
There's never a problem with testing something again @shartley76, feel free to give it a go and see. |
In the latest version, this issue is not reproducible. I am not able to replicate it at my end. Below are my configurations.
I tried to upload a file with the size of 800MB and able to upload it. May I have more details on configurations? |
@shartley76 keep in mind that the latest version of the API is 2.2.2, not 2.0.21 🙂 |
Side note, we are using Docker and the last available release for that was 2.0.18. We have been manually uploading any files over a certain size and editing the database since we are stuck on 2.0.18. |
Docker refactor is in work now so that you can get on the latest version, @WoLfulus should have an update soon! |
Bug Report
I am running Directus app and api on Docker using the latest images. I have created my own dockerfile to move custom php settings to allow large file uploads (instead of the default 2Mb).
modified php settings
upload_max_filesize=500M
post_max_size=500M
memory_limit=500M
max_execution_time=1000
modified NGINX conf
client_max_body_size 500M;
However with the above settings I can upload a 20Mb video file, but any file that is just over 100Mb in size always returns an error (even though it should allow up to 500Mb).
I cannot seem to find any log information to determine the cause.
Steps to Reproduce
Expected Behavior
File that is >100Mb but less than 500Mb should be uploaded successfully to file library
Actual Behavior
The large file returns a server error (500) once the upload has completed. See attached image.
Other Context & Screenshots
Technical Details
The text was updated successfully, but these errors were encountered: