Share
vasiliauskas.vyt 8:20am, 10 September 2020
API randomly throwing oauth_problem=signature_invalid error. What I might be doing wrong? It works but only sometimes throws this error.
esby.photo 1 year ago
is it really random or reproductible ? (will a failed request success later ?)
Xenedis 1 month ago
vasiliauskas.vyt:
API randomly throwing oauth_problem=signature_invalid error. What I might be doing wrong? It works but only sometimes throws this error.

I am also experiencing random occurrences of 'signature_invalid'.

In looping over a list of ten collections, my authenticated call to flickr.collections.getInfo fails randomly.

In a given run, a call might fail for one particular collection ID, but it can be successful for the same collection ID on a subsequent run.

It is seemingly random.

As an experiment, I tried using the same nonce and timestamp for all ten API calls.

Failures still occurred, but the failures happened on the same collection ID on every run.

I really cannot pinpoint the cause of this problem.
admin
Sam Judson 22 days ago
Most cases I have seen of these seemingly 'random' error is when one of the parameters is being encoded badly when put in the signature. The encoding is 'almost' URL encoding, but not quote.

oauth.net/core/1.0a/#encoding_parameters

The following characters must NOT be encoded: '-', '.', '_', '~'

Also, a space is always encoded as %20 and never as "+" which is allowed in URL encoding.

Finally the last thing to check is when appending the signature to the URL you are using actual URL encoding (i.e. not the same as above).
Xenedis 22 days ago
In my case, I am iterating over a list of collection IDs, and calling flickr.collections.getInfo.

The URL encoding is done using PHP's oauth_urlencode().

The only parameters which change on each iteration are the collection ID, timestamp and nonce, which is a randomly-generated eight-digit integer.

I am not sure how the encoding could be right sometimes, and wrong sometimes, as it is the same code being executed for each collection ID; the encoding should be either always right, or always wrong.
admin
Sam Judson 22 days ago
I agree, sounds unlikely that it is the signature generation that is wrong.

Are you using the same method (oauth_urlencode) to add the signature to the URL as well?

Would it be possible to provide some examples of the signature when it works, and the signature when it doesn't work.
Xenedis Posted 21 days ago. Edited by Xenedis (member) 19 days ago
Firstly, here is a detailed explanation of what I am doing.

I start building the base string by concatenating the following three strings:

GET&
https%3A%2F%2Fapi.flickr.com%2Fservices%2Frest%2F
&

The URL above is the output of passing the following URL to oauth_urlencode():

'https://api.flickr.com/services/rest/' (without the apostophes)

To complete the base string, I use a loop to append to the base string all of the parameters and their values, in alphabetical order, encoding each with oauth_urlencode().

The syntax of the first parameter is:

parameter=value

The syntax of subsequent parameters is:

&parameter=value

The parameters and values are as follows:

api_key = my Flickr API key
collection_id = collection ID
format = json
method = flickr.collections.getInfo
nojsoncallback = 1
oauth_consumer_key = my Flickr API key
oauth_nonce = randomly-generated eight-digit integer
oauth_signature_method = HMAC-SHA1
oauth_timestamp = current UNIX timestamp
oauth_token = my Flickr access token
oauth_version = 1.0

In parallel, I am using that same loop to build the query string which will form the API call.

The only difference is that the first parameter begins with a question mark, as it is used for a URL query string in this instance rather than a signature.

The next step is to build the signature.

I do this by generating a SHA-1 hash of my base string, with the key comprising a concatenation of my Flickr API secret, an ampersand and my access token secret.

This is all Base64-encoded.

Finally, the API call itself consists of the following strings concatenated:

'https://api.flickr.com/services/rest/' (without the apostophes)
Parameters (eg, ?parameter1=value1&parameter2=value2)
&oauth_signature=
My signature

Here is the output from running ten collection IDs through this logic:

2265452-72157679383068781 - Failed - signature: xyzsK+z6p23SVbSuD+uYF7rO71U=
2265452-72157677820508924 - Failed - signature: TQF1AYgj+Hu3LHiOJw7t+79VGGU=
2265452-72157679382469071 - Passed - signature: hM3EITDVY2nUT0LgI0wD0QQklGk=
2265452-72157679384084231 - Passed - signature: /xf6g6DjMS7oHswAMLoORTd7Cxg=
2265452-72157677821348434 - Passed - signature: mBvdGw3n235PRDhwMUNj10Jbld8=
2265452-72157677821548284 - Failed - signature: 2rhRRCHCBfMD+/SooWJkkzYyDlI=
2265452-72157682522469424 - Passed - signature: WJVYY1f0w4kNPcdefHBT3gc/1lI=
2265452-72157679383933051 - Failed - signature: efStqmKaVxEn+afDxF0SfWb0zk8=
2265452-72157677821941094 - Passed - signature: wzbS5dy8FAmY9QQYUwfMGlVtkdY=
2265452-72157677820315054 - Passed - signature: /6wahXQtlTyNEpKlve7lc97GoS8=

When I ran the same code again immediately after the first run, the following output was the result:

2265452-72157679383068781 - Failed - signature: MuTs/VVUJXjNchCxw+WAg1P9n5A=
2265452-72157677820508924 - Failed - signature: xFfjSDstyh72OvMclGy+tCAMQ+w=
2265452-72157679382469071 - Passed - signature: dWJgyEWiMtdsHaSun1pYepysICI=
2265452-72157679384084231 - Passed - signature: Bc0ecYm2NQYWEq7GDGFlP/5vSXQ=
2265452-72157677821348434 - Passed - signature: VnuLqaEVTO6CzbxfPygCjSFJ/ww=
2265452-72157677821548284 - Passed - signature: fRt5Q9qHElaa7VjphdXEeUkxgSM=
2265452-72157682522469424 - Passed - signature: 8rhz9H7NJ/SLYvv4LWs6HTJbAHA=
2265452-72157679383933051 - Failed - signature: B4V5QjxP9/RHmTVKR4eg1+lcFOU=
2265452-72157677821941094 - Failed - signature: M+FgQ8twQeL0RVKLR5k4Gi0TAjk=
2265452-72157677820315054 - Passed - signature: HbfzODgi7lytxTAMXNiDhow5hlA=

As you can see, on the second run, the sixth and eighth collections failed, whereas they passed on the first run.

The number of failures varies, as does the collection ID which results in a failure.

On a third run, there was only one failure, and happened to a collection that passed during the first two runs. Here is the output:

2265452-72157679383068781 - Passed - signature: hGzG7YjWUTy9IfwPKOYE04xDCDg=
2265452-72157677820508924 - Passed - signature: MjD9JdB/94IulMdvjBNsULu80bM=
2265452-72157679382469071 - Passed - signature: ObRNTc/ks4mi7DfBgRnV4MBNFqw=
2265452-72157679384084231 - Failed - signature: gxj64jd0ZS3894+5o5b1GEi8te4=
2265452-72157677821348434 - Passed - signature: Y8X9m1KGkhoqsaHrRS1WVAmX6F4=
2265452-72157677821548284 - Passed - signature: qe938sYwxSERNUDkguCxL7G4cA4=
2265452-72157682522469424 - Passed - signature: sNdQ8pQqFsFCJoY0qzpcVG/NfUg=
2265452-72157679383933051 - Passed - signature: gG2nXI/IBYpV3fWIIhpKUt3OvK0=
2265452-72157677821941094 - Passed - signature: /k7Bt15WGVy0yOdsZ7z5ghhlGyQ=
2265452-72157677820315054 - Passed - signature: gV8N4TnhdkyrZgoYUHm7VMfFPWg=

This is really baffling.

If I had made some sort of error in my approach, I would expect to see the same results for every API call; but this is not what is happening.

The number of failures, as well as the collection ID on which the call fails, varies.

The only consistency I saw during earlier runs a few weeks ago was failures on API calls for specific collection IDs when I experimented by using the same nonce and timestamp for every call.

Obviously in that scenario the signature for each call would still be different due to the collection ID being unique; but unfortunately I am equally unable to explain those results.

One important point is that, from memory, I also experienced issues with invalid signatures during the initial token generation/exchange phase. Sometimes, it worked; on other runs, it did not.

It does lead to me to wonder if there is something wrong in my approach; but what I cannot explain is why it sometimes works, and sometimes fails.

I wonder whether or not I should try another API call (on, say, image IDs) to see if I experience the same results.
Xenedis Posted 19 days ago. Edited by Xenedis (member) 19 days ago
Xenedis:
I wonder whether or not I should try another API call (on, say, image IDs) to see if I experience the same results.


I decided to try another API method.

This time I used flickr.photos.getSizes, and passed a list of 20 image IDs to it.

Each call was successful, and on each of the three times I ran through the 20 images, there were no failures.

When I tried using flickr.collections.getInfo again, I experienced random failures.

This suggests to me that my code for generation of the base string and call signature is correct.

This raises some questions, though.

Could flickr.collections.getInfo be buggy?

Could the presence of a hyphen inside a collection ID intermittently cause problems with URL encoding? I would not expect so, because the Flickr access token also has a hyphen in it. Additionally, PHP's oauth_urlencode() and urlencode() functions do not encode hyphens.

I still cannot see why calls to flickr.collections.getInfo randomly fail.
admin
Sam Judson 17 days ago
Every single failed signature has one thing in common - they all contain the "+" symbol, and non of the successful ones do.

So I would suggest you look very carefully at the URL encoding you are doing and make sure you are encoding the signature in the URL correctly. As I mentioned above in a URL a space can be encoded either as "+" or "%20", so the "+" would definitely need to be encoded.
e.g.

gxj64jd0ZS3894+5o5b1GEi8te4=
gxj64jd0ZS3894%2B5o5b1GEi8te4%3D
Xenedis Posted 17 days ago. Edited by Xenedis (member) 15 days ago
Sam Judson:
Every single failed signature has one thing in common - they all contain the "+" symbol, and non of the successful ones do.


That is an interesting find, and something I had not observed.

However, there remains the crucial point that the encoding method is the same on every iteration. It is literally the same code being executed.

Sam Judson:
I would suggest you look very carefully at the URL encoding you are doing and make sure you are encoding the signature in the URL correctly.


I am using the same encoding method for every iteration.

The only parameters that change are the nonce (a random eight-digit integer), the timestamp (an integer) and the collection ID (which does not contain any spaces or other characters which need URL encoding).

Every other snippet of data which makes its way into the base string and signature is static.

Sam Judson:
As I mentioned above in a URL a space can be encoded either as "+" or "%20", so the "+" would definitely need to be encoded.


I can see how that would be a problem if my URLs had spaces in them, but not one snippet of data contains any spaces.

Given that none of the strings which comprise the base string or subsequent signature contains a space, how could a space possibly end up in the signature and be mistakenly encoded as a plus symbol?

Again, I would expect to see consistency here (given that the same encoding is performed within a loop), and that is not what is happening.

I really cannot see what I could possibly be doing wrong; but at the same time, I cannot reasonably suggest that PHP's oauth_urlencode(), hash_hmac() or base64_encode() functions are somehow getting it wrong intermittently.

The cause of these random failures is elusive.

If you would like me to provide you with my code, I am happy to do so.
Xenedis 9 days ago
At this stage, I have hit a wall with signing API calls.

PHP's base64_encode function sometimes results in a plus symbol being a part of the signature, which is not unexpected behaviour, given that the plus symbol is one of the valid characters in that numbering system.

It appears that the Flickr API has a problem handling plus symbols in signatures.

Calls using signatures with plus symbols fall on every occasion, whereas calls using signatures without plus symbols succeed on every occasion.

I have seen examples of user-created functions which replace the plus symbol with another symbol (and even PHP's base64_encode() documentation page contains one in the user contributions); but I have tried this approach, and it does not work.

I really do not know how to solve this problem.

What I do know is that there are no spaces in any of the data which goes into the base string or API call URL.
admin
Sam Judson Posted 8 days ago. Edited by Sam Judson (admin) 8 days ago
I'm not sure what the solution is, but I need to reiterate 2 things.

You are encoding each parameter and value twice, once to add it to the base string which is passed into the signature generation, and then again when you add it to the URL (the thing you send to Flickr).

Once you have the signature you need to base64 encode it (so its a string) and then URL encode it before you add it to the URL you send to Flickr, e.g.

// calculate binary signature
$signature= hash_hmac("sha1", $baseString, $secret, true);
// base 64 encode, then url encode to add to the URL
$url .= "&oauth_signature=" + urlencode(base64_encode($signature));
Xenedis Posted 8 days ago. Edited by Xenedis (member) 8 days ago
Sam Judson:
// base 64 encode, then url encode to add to the URL
$url .= "&oauth_signature=" + urlencode(base64_encode($signature));


Finally, the problem has been solved.

The problem was that I was not URL-encoding the signature before finalising the API call URL.

That explains why the generated signature worked as long as there were no plus symbols within it.

Having run my code a few times now, every API call succeeds.

Thanks for pointing out what has eluded me for weeks.

Looking back at this thread, you did raise it a few times, but somehow I did not recognise the significance of it or otherwise connect the dots.

In hindsight, it now seems so simple.

Hopefully this thread will help other people who have encountered, or will encounter, problems with invalid signatures.
Groups Beta