General overview
What is happening?
When the Prohashing project was conceived on December 23, 2013, it was projected that the potential market for a coin-switching mining pool was very small. Previous business attempts in other industries in 2009, 2010, and 2012 had failed due to being unable to capture a significant market share. For success, it was thought that the project would have to produce a mining server that could handle a reasonable number of miners, but that profitability would only result from a heavy focus on advertising and offering a large portfolio of services, each of which could attract a small number of customers.
For a reason that still hasn't been able to be fully determined, Prohashing suddenly had an influx of customers in May-June 2017 and gained 20 times more market share than the most optimistic projections, despite not having performed any advertising in months. The dramatic difference in required a change in direction, where all resources were focused in improving performance. While performance has been improved over 1500-fold from the original capacity, it is just barely sufficient for the number of customers. Additionally, we had not anticipated so many customers and are now scrambling to find people experienced in business, taxes, law, and other areas where we have no expertise.
Directing all resources at improving performance led to a deterioration of other services due to a lack of manpower. As a result, the number of customer support tickets increases. Since all customer support tickets need to be answered, temporary fixes like balance corrections take priority over permanent fixes to the problems that generate the tickets, like mining server bugs. As a result, most effort now flows to temporary fixes to individual customer problems, and additional tickets are generated when the problems recur because no manpower can be diverted from answering the tickets to fixing the problems permanently.
What is being done about these issues in the long term?
The project underwent an evaluation to determine its future viability by updating the original projections based upon current trends, and a decision was made on September 2. The results of the evaluation, when it is announced, will determine how the current problems are addressed.
What is being done in the short term?
In the short term, we sometimes close the project to new registrations in order to reduce the number of support tickets. While the current issues are not due to system overload, the largest number of support tickets understandably come from new customers, who are unsure about pool features. Reducing the number of support tickets will provide more time to fix issues permanently.
Currently, registrations are open.
Summary of issues
What issues are currently the most problematic?
The primary problems right now are connectivity issues caused by the DDoS protection. The network issues cause some miners to be unable to connect. Additionally, they cause the WAMP server to disconnect from the mining server, resulting in inaccurate share data. The DDoS attacks in June forced the project to implement a complex VPN setup that is difficult to maintain and which restricts functionality. A solution to this problem is to purchase an enterprise-class Internet connection, which requires a multi-year contract valued at $80,000, which is only viable in the case of expansion.
Performance issues, and bugs created by fixing them, are another problem. Every time a performance improvement is implemented, bugs caused by the small scale of the testing environment need to be fixed over the subsequent weeks. Then, the system is almost immediately overloaded with new miners, and new improvements need to be introduced, repeating the process.
Finally, the mining server is currently running a suboptimal coin assignment algorithm because running the optimal algorithm we were using before June would result in CPU overload. If dramatic expansion is chosen and the coin assignment is parallelized, the optimal algorithm can be used and stale shares can be reduced by 5-10%.
What caused the issues?
Most of the current issues were present since the system was originally introduced. They were uncovered now because the number of unique miner configurations has increased. While some customers have surmised that one feature or another has caused issues, introducing features like x11 mining is not responsible for the current problems. Instead, what happened was that x11 mining and similar new features drew in customers with different needs and mining patterns, these configurations had never been anticipated before, and therefore they could not have been tested. The new configurations trigger unforeseen issues that sometimes affect other miners.
It has become apparent that future issues could be prevented by investing a large amount of money in a duplicate mining farm, so that more configurations could be tested before release. This option is only feasible in expansion.
Are the fixes to most of the issues known?
Yes. We know exactly how to fix most of the issues, so it isn't necessary to speculate on the forums what the problems are. The only limitation is manpower to execute the fixes, and we cannot move forward on that until a lawyer completes research on the impact of any future regulations on the profitability of the business.
Are cloud mining services responsible for the issues?
Not currently. Cloud mining service users caused issues in July when some miners submitted extremely low difficulty shares using cloud miners, overloading the system. Since then, the performance to process shares has been dramatically increased. Most of the current issues are caused by bugs introduced by the performance improvements.
The cost of rentals at cloud mining services does appear to be affected by our system, but our primary responsibility is to provide good service for our customers. The issues at Prohashing are no longer related to usage of cloud mining hashrate, and issues with cloud mining services are the services' responsibility to address.
Why don't you buy more of (some item) to address the issues?
The issues are software-related, not hardware-focused. Insufficient parallelism in the system has caused CPU overload of many services. We have developed an architecture that we believe can resolve the issues, but implementing it is expected to take about 1000 hours of work. Fortunately, the benefits of the new architecture can be felt as each piece is developed; the share processing piece has already been parallelized and as a result the database rarely if ever gets behind on share insertions anymore.
The next piece to be parallelized is coin assignment, which is currently causing a large number of stale shares because coin assignment is unable to be performed in real time. While we have made progress on parallelized coin assignment, we have decided not to release it because it will be the most significant change in the project's history, and it will need constant monitoring. The probability of a bug to occurring on a Monday morning, when it cannot be repaired, is too high.
Why don't you move to Amazon or another shared hosting service?
We made a decision that we must own all physical hardware for our system. Because up a quarter million dollars of cryptocurrencies is stored on these servers and paid out daily, they are extremely vulnerable to attacks. Common attacks to past pools have included people calling hosting companies and pretending they are the site owners, and system administrators pulling disks from the hypervisors to copy wallet keys.
By owning the hardware on which the system runs, nobody but us can make server configuration changes, so an entire class of attacks is eliminated.
A few users have offered to provide hosting in the past in an effort to solve issues. While we are very thankful for their kind offers, we hope that everyone can understand that we must maintain sole physical control of all systems. Plus, since the system is software-limited, a faster connection or more memory at a shared hosting service won't resolve the current issues.
Why do my shares seem out of proportion to my earnings?
One of the issues plaguing the system is dropped shares. We are striving to try to determine the causes of the problems. Chris automatically corrects these issues when they are detected, but we cannot promise that all corrections are 100% accurate because many times the information needed for the corrections has been lost. However, share corrections have not been issued since September 9, so miners experiencing low earnings after that date should investigate other causes.
Share corrections are not issued for periods when miners cannot connect to the pool. In that case, you should configure your software to use a backup pool.
The evaluation
What was being evaluated?
The evaluation was aimed at determining the profitability of the project under an optimistic scenario (the status quo), an average scenario (scrypt returning to its breakeven profitability of 0.1 cents), and a pessimistic scenario (like what occurred in January 2015 when most mining equipment turned off.) Each of these scenarios is being evaluated under a number of spending and manpower levels.
In addition, the competition is being analyzed to determine the likelihood the project can compete in the marketplace. Finally, a lawyer has been
contacted to determine what, if any, legal issues could arise.
What are the critical factors for profitability?
The critical factor for profitability is whether BCC and ETH remain as proof of work, or go proof of stake, as these coins are currently 75% of all profit potential. Other profitability factors include whether the Bitcoin Core forces more forks of the bitcoin blockchain, and whether DASH and LTC maintain prices well above historical values.
What concerns are holding the project back?
The critical factors for expenditures include the cost of the duplicate testing mining farm, the projected expenses of protecting against DDoS attacks, the risks of legal problems due to the new field, and the low margins inherent in mining.
In particular, protection against DDoS attacks is projected to cost as much as $3,000-$10,000 per month, and successful attacks result in tens of thousands of dollars in lost profit due to downtime. DDoS attacks are by far the largest expense and the issue that causes the largest concern in these calculations because it has not yet been able to be proven that attackers could not bankrupt the business either purposely or accidentally.
Legal problems are another risk. We already have enough money to survive for a long time without work. If we accidentally violate a law we didn't know about, we could end up paying more in fines than we had earned. We want to follow the law and need to be certain that we can do so, but since this is such a new field, there is little precedent and it takes time to perform research.
Mining has low margins, and one of the issues with expansion is that it isn't clear whether these low margins could support a higher quality of service than paying additional employees would allow.
Another concern is whether the system could ever be parallelized enough to support the performance necessary to obtain profitability to offset the costs of expansion. There is a school of thought that it may simply not be possible to create a very large multipool system with current hardware technology. This is a factor in the decision, since the only way dramatic expansion can be profitable is if many more miners are able to join the project.
Finally, research has found that many of our competitors are anonymous entities who may be evading taxes. Since we are not willing to break the law or operate anonymously, we are at a significant disadvantage competing against such companies. We need to make 2.04 times their profit to make up for the 52% we pay in taxes.
Is the project currently profitable?
Yes, if additional regulations are not imposed. However, scrypt profitability is far above the breakeven point. An efficient market should drive scrypt profitability down to no higher than 0.3 cents (especially in the winter, when heat has value), because that is the level at which miners can just barely pay for their electricity with the newest miners. We believe that it is very unlikely that the current profitability will last for very long since mining equipment is being turned out at a fast rate.
What were the objectives of this evaluation?
The primary objectives of this evaluation were:
- To improve the level of service to customers by reducing bugs
- To reduce the number of hours worked on the project to a long-term manageable level (the current target is a reduction from 70 to 60 hours per week)
- To take few or no risks with our existing savings (such as avoiding legal risk)
- To maintain the ability to live away from a major city
What other considerations were there?
Other considerations in the decision, besides profit potential, were:
- This project is less interesting than Steve's current position, and that position has significant responsibilities with many subordinates that cannot be wrapped up simply without careful planning
- The Affordable Care Act, which protects people with preexisting conditions like we have, is in jeopardy from Republican actions, and Trump recently weakened the Act with an executive order
- This project requires more work, stress, and risk than a standard job, does not provide benefits, has great legal risks, and therefore needs to pay far more than a standard job to be worth it
- Shady people are involved in cryptocurrency and it is possible we could all be made to look like fools tomorrow when they collapse the industry out from under us
- On a philosophical level, the rise of ETH has already brought us enough money to satisfy our modest living costs, and it is stupid to work so much to do things like buy two houses or fly first class. We spend so little money because we have no time to buy anything and can't think of anything to buy that would improve our quality of life anyway. We understand why those who are struggling would say that this decision is a no-brainer, and we agree we would also be more risktaking if we were extremely poor and had less to lose.
What options were considered?
Four options were originally considered, but two of them were dropped immediately. Maintaining the status quo was dropped because we believe that even though profit could be made, doing so would not provide the sort of customer experience we want to provide. A small-scale expansion with just Chris and Steve working 60 hours per week each was also dropped, because it was determined that it would not be possible to earn enough profit to justify the risk of full-time work when only two people were working.
The two remaining options considered were to dramatically expand, or to downsize.
What would dramatic expansion involve?
Dramatic expansion, which would cost about $400,000 in the first year, would involve full-time work, hiring one or more employees, seeking legal advice, and investing money in a full-scale duplicate environment of the production system, including a variety of different miners in a mining farm. One employee would be tasked almost solely with creating, maintaining, and using the test mining farm. The test system, which is estimated to cost about $50,000, would significantly reduce the number of future production bugs because a greater number of miner configurations would be present on the test farm.
Dramatic expansion would involve the purchase of enterprise class Internet service to significantly improve reliability and uptime. This $3000/month service can only be justified with significantly increased profit.
Dramatic expansion would result in one employee focusing full time on performance improvements and bugfixes to the mining server, one employee answering customer support requests and improving the trader, one employee maintaining the test farm, and possibly other employees improving the website and dealing with bureaucratic work.
What would downsizing involve?
Downsizing, which would be decided upon if it is determined that the current situation is likely to be a bubble or if the pool would not be profitable enough to support additional employees, involves limiting server usage to the maximum supported, freezing new features, spending all time fixing existing bugs, and not expanding into Dagger-Hashimoto and SHA-256. The goal would be to produce a small, but profitable, pool that would provide a second source of income until someone willing to risk more money eventually produces a better product and we are unable to compete.
How can dramatic expansion be profitable in such a low-margin business?
Our initial evaluations, which were completed August 25, have indeed demonstrated that the dramatic expansion option loses -$500,000/yr in the pessimistic scenario under existing fee levels with the existing situation. The largest expense in this scenario is mining server bugs, which cost -$1.059m/yr.
We are currently evaluating the impact of an increase in fees, perhaps to 10%. While that sounds like miners would earn 5% less, mining server bugs are currently estimated to reduce profitability by 10% themselves. Counterintuitively, it is more likely than not that a 5% fee increase to pay someone to take care of bureaucratic tasks so that Steve can fix mining server bugs would result in a 5% profitability gain for miners.
Why do you keep talking about bubbles?
There have been at least five bubbles in bitcoin's history. During the upswing in the bubbles, most community members believed that high prices and cryptocurrency usage would continue indefinitely. There has been a trend upwards, but during the down cycles mining becomes very unprofitable. While it isn't clear when the current bubble will end and how the industry will stabilize after the fall, there is no reason to suggest that this time is any different.
What can you say about what the evaluation revealed?
Under the pessimistic scenario, which involves a reduction in LTC price to $20, a fall in market share due to BCC going POS, and the inability to expand into ETH due to ETH going proof-of-stake, net profit is just $142,928.11/yr - insufficient to risk expansion. Under the very optimistic scenario, which involves the status quo continuing for a year, the project would earn several million dollars in profit.
Therefore, we determined that the future of the project depends almost entirely on external factors out of our control. We determined that we needed to perform additional research to determine the probability of these external factors occurring.
What was the timeframe on the evaluation?
The evaluation was delayed due to support tickets generated by current bugs, but it was completed and a final decision was reached on September 2 at 3:00pm EDT. Now, we are obtaining legal advice. If the decision is to dramatically expand, we anticipate another three weeks to wrap up other commitments and hire personnel, for a target start of late November. If expansion is chosen, significant progress in resolving bugs and improving performance would be expected by Thanksgiving. In the downsize option, progress will still accelerate because of attrition due to closed registrations resulting in fewer support tickets and more work on bug resolution.
When will the decision be announced?
A decision was reached, but we can't announce it yet until all possible legal issues have been addressed. There is no ETA as to when we will be able to announce the decision because the lawyer is still performing research to rule out any potential legal issues. Prohashing is not currently facing any legal actions, but this evaluation must be completed and we must implement any findings before we finalize our decision.
How can I invest in Prohashing?
The projections indicate that expansion is definitely not worth the risk if our profit is diluted by shareholders. Therefore, we are not accepting investments and plan to retain 100% equity in the company.
Hiring and jobs
Is hiring people being considered?
Several of the possibilities included the possibility of hiring employees. The type of employee to be hired was being evaluated as well - while a customer support representative or accountant is the greatest need, a developer might be able to do those things in addition to development.
Why isn't someone being hired immediately?
There are a number of reasons. First, we can't hire someone until someone is available to supervise full time. Most employees want to work on weekdays and we do not believe that a new employee can adequately learn alone during the weekdays when the supervisors are working on the weekends.
Second, the first employee needs to be someone who has management potential because we want that employee to grow as a top leader within the company and supervise other employees who come afterward. The characteristics we are looking for in a first employee, therefore, are not simply a developer who can write code. We are more interested in someone who has the "soft skills" necessary for leadership, even if that person may be weaker technically. We would like that person to be local so that when future employees are hired, we can talk to him or her in person as (s)he supervises potentially remote employees.
Hiring someone requires a significant commitment. It places the responsibility of making sure that the person has a stable job on the employer. We are not willing to ask someone to make a life change without being fully committed ourselves to his or her success.
Additionally, hiring someone would not address the current problems because a significant amount of training would be required. The training would actually reduce productivity in the short-term by taking Chris away from direct tasks. Furthermore, since some issues cannot be parallelized, placing a less experienced person on a serial task will slow it down, not speed it up.
Can I help you part-time on the weekends to make some progress on things?
We are not hiring for part-time work. Past experience with part-time employees has demonstrated that the project is so complex that too much background knowledge is necessary to accomplish even minor tasks. The part-time employees were left behind and were spending almost all their time on training as everyone else continued to do accomplish tasks while they weren't working.
Can I apply for a full-time developer position?
Unfortunately, developers are not the skillset we need. We've already identified one developer if the evaluation yields that we should expand. However, if the evaluation calls for more than one employee, or if additional developers are required in the future, feel free to create a support ticket with your résume attached. It will be recorded in a database for future consideration of full-time employment.
If you are a certified public accountant in the cryptocurrency area, we are accepting help; please open a support ticket for more information.
What can I do to help?
While we aren't ready to hire people yet, there are ways that can provide significant help.
- Do not submit support tickets for balance corrections or payout issues unless significant time has passed and you are sure that the problem is still present.
- Assist users where possible in the forums.
- Back up your two-factor authentication keys and your passwords. With closed registrations, loss of a two-factor authentication code now means that you cannot mine until registrations reopen.
Support tickets
How many support tickets are being handled?
The number of support tickets submitted per day averages around 50. Tickets generally back up during the week and then are addressed during the weekends. During periods with smaller numbers of bugs, many fewer tickets, as few as 10 per day, are submitted.
Which issues generate the most tickets?
The largest number of tickets are generated by new customers, which is to be expected. Common questions include how to mine and misunderstandings of the 24-hour payout system. One of the reasons we want to limit registration is because we don't feel that we are able to provide the level of support that these new customers deserve.
How are support tickets answered?
Support tickets are answered from oldest to newest according to last reply, so that people waiting the longest are addressed first. When Chris or a customer replies to a ticket, it moves to the back of the queue so that other tickets which have not had a reply are answered first.
Some customers have replied to tickets asking questions such as "has this been resolved yet?" Since tickets are answered in chronological order, replying to a ticket that has not been answered causes it to be delayed until the hundreds of other tickets awaiting a reply are answered first.
Are all issues addressed?
Chris guarantees a reply to all tickets, but can't address all issues. Some issues, like the effects of the pool on the EQT coin, require significant changes to the mining server code that would require a day or more of work and which cannot be prioritized above fixing balance issues and other bugs. We apologize for this and these problems are one of the critical reasons why we want to change the way the project is run in the future.
Why is it taking so long to receive a reply?
90-95% of the support tickets that are received result from issues that are already known about, like balance corrections. In the past, we replied to support tickets as soon as they were received, but we found that immediate replies took away time from fixing the issues before they generated more tickets.
We are currently in reduced support mode, in an attempt to free up time to solve the core issues that repeatedly result in the most tickets. Issues are being addressed, and then all the tickets pertaining to each issue will be replied to at once. We sincerely apologize for the delayed responses, but we guarantee that we will respond to every customer eventually and one of the reasons we are considering expansion is to provide better customer service.
At this time, it could take up to one week to receive a final response to some tickets. The response time is down from two weeks in August due to fewer tickets being submitted as a result of fewer bugs.
Could you work with NiceHash to improve compatibility?
Prohashing does not have an association with NiceHash and does not guarantee compatibility with NiceHash.
If you want to write bots for NiceHash or rent miners at NiceHash, there is no prohibition from doing so, but you will need to contact their support if you encounter issues with their service.
My inbox didn't receive an E-Mail notification from the support ticket system when you replied. What happened?
Some ISPs are returning error messages stating that "some part of your ISP's network" is on some sort of blocking list. Your E-Mail provider is deleting the messages. Microsoft is one provider that uses these blocking lists.
Contact your ISP and ask them to allow E-Mail from prohashing.com addresses in order to get around this problem, or switch E-Mail providers.
Why is my ticket about a forgotten password taking so long to receive a reply?
Since we are overloaded with customer support requests, we have decided to prioritize password reset requests at the lowest priority. While the customer is always right and every ticket is important, in a time of limited resources we need to fix the issues where the pool is at fault first.
We are continuing the work down the ticket queue and plan to respond to all password reset requests eventually. We apologize for the delay.
Should I submit a ticket for balance corrections?
No. Chris reviews balances at the end of each day to determine if any shares were lost, and makes corrections independently. Submitting a ticket only slows down the correction process because all tickets have to be answered.
Should I submit a ticket for payout issues?
Usually not. First, payouts are only guaranteed to occur the next day, and many tickets are submitted before 24 hours has elapsed. Second, Chris evaluates all payout issues every day and independently addresses the coins with the most value that failed payout the previous day. You should only submit a ticket for payout issues if significant time has passed and the issue has not been addressed independently.
Thanks for reading! We will update this FAQ every weekend for the next few weeks with the latest information. The "last updated" date below documents when it was last updated.
While I do not have a degree or professional experience programming, I do have a background in it and still know my way with the old binary digits. Plus, you wouldn't have to pay me, as long as lowered profitability due to intentionally unprofitable testing is compensated. There may also be others in a similar situation who could add to your test pool.