It may contain salt. So later it will result in shrinkage and cracking. And it will also not result in same strength as river sand gives.
Definitely the salts adsorbed (dissolution might not actually occur) on the sand are going to affect the durability in reinforced concrete structures but in plain concrete or mortar for plaster, it might not be that detrimental. It is not that hard to get rid of salts adhered to sand particles.
However, apart from salts, there are several factors that hinder its use, like aggregate shape and size. Sea sand, generally tends to very fine and rounded, which is not particularly advisable when designing mixes. A kind of blending is required with coarse and angular aggregate to develop the rheology and microstructure desired.
However, when the economy demands, it is used. Practically speaking, the aggregates are good to use as long as they are in compliance with respective codes of practice (like ASTM C33). If the aggregates make it through all the tests and clauses, then you can officially use the material in construction.
Edit: The possibility of the use of crushed gravel.
Crushed gravel would be siliceous in nature and could be a decent alternative. However, the process could be slightly demanding, since crushing operations are usually performed on larger boulders to “crush” them to required sizes and distribute for use.
There could also be an issue of salts. In my post, I said that they could be physically removed, which might not be completely true.
You see the water (which usually has dissolved salts) that is present will be absorbed (absorption capacity of aggregate) and in the process, the salts may get deposited in the porous microstructure of the aggregate, which ultimately cannot be removed by physical treatment completely.
So a careful investigation is recommended that is more than just tests on aggregate since the combination of water (in concrete) might release the salts back into solution.